#linuxcnc-devel Logs

Mar 03 2019

07:08 AM whitequark: rmu: afaict i'm interested in get_pos_cmds() and output_to_hal() here https://github.com/LinuxCNC/linuxcnc/blob/f36ddbe0e0cc43adf379aa430114f67ee26efa30/src/emc/motion/control.c#L285-L288
07:10 AM whitequark: which do the job of calling the tp and writing to hal
07:34 AM rmu: whitequark: sorry, i'm currently cleaning up / reorganizing the workshop, AFK most of the time
07:35 AM rmu: whitequark: if you want to do all real time stuff in FPGA you probably want to extract the segments (lines and circles) out of the TP.
07:36 AM rmu: usually, the segments are fed into one of two functions (ramp accel and trapezoidal accel), that would be the parts that have to be run under realtime constraints and generate position/velocity updates
07:36 AM rmu: rigid tapping will need some extra logic
07:37 AM whitequark: ohh i see what you mean
07:41 AM rmu: in my experiments (see here https://forum.linuxcnc.org/38-general-linuxcnc-questions/34666-c2-smooth-velocity-profile) i stumbled upon some strange situations where the TP changed segments at a very late stage and/or generated segments that did not meet exit velocity constraints, with my hacked-together acceleration profile these situationes generated acceleration violatins
07:41 AM rmu: unfortunately i didn't have time to dig deeper and clean up my code
07:42 AM whitequark: rmu: ok, so i'm thinking of a kind of "multi-phase" experiment
07:42 AM whitequark: phase 1 would be replacing parport, phase 2 would be replacing stepgen, phase 3 would be taking segments off the TP directly
07:43 AM whitequark: actually i'm wondering if an USB LPT adapter that actually works for linuxcnc would be a useful product on its own right
07:44 AM whitequark: it's simple enough you could do it with a toner transfer PCB process even
07:44 AM whitequark: unsure if there's demand but i know exactly how i would make it
07:47 AM whitequark: oh interesting http://ecklersoft.com/pymini.html
07:47 AM rmu: replacing parport and stepgen is easy, very easy on ARM boards like raspberry pi.
07:48 AM whitequark: yeah i know, but the point is not to use some board with cheap GPIO (i know that's easily done)
07:48 AM whitequark: but to validate the realtime usb stuff without spending too much on custom hal modules
07:49 AM rmu: it should be possible to talk USB from hal module with rt-preempt kernel
07:51 AM whitequark: the pymini person extracted the tp from linuxcnc in a rather ad hoc way, but they don't need any rt bits at all. hmmm.
07:51 AM whitequark: i can see the appeal of that but i also want to reuse linuxcnc
09:26 AM rmu: i think the parts of the trajectory planner that depend on realtime need to be fixed anyways...
09:40 AM pcw_home: Some parts will always need realtime though (spindle synced motion etc)
09:41 AM pcw_home: adaptive feed
09:48 AM pcw_home: similar situation with hal, if you dont implement hal or something like hal in real time, things like plasma THC are awkward
09:50 AM pcw_home: a buffered system like mach (non real time host talking to small real time motion engine) means that any real time feedback needs to be implemented in the attached real time engine
09:51 AM rmu: i think that is what whitequark is contemplating
09:51 AM whitequark: yeah
09:51 AM pcw_home: Yeah its nice in that its host agnostic but not so great for more complex controls
09:51 AM whitequark: when you're talking about plasma THC, do you mean e.g. "torch on/off" commands?
09:52 AM rmu: spindle synced motion in combination with feed- and spindle-speed-override quickly gets awkward to handle
09:52 AM pcw_home: no, real time Z axis motion based on plasma voltage feedback
09:52 AM rmu: you want to control torch height (via z-axis) depending on arc voltage
09:53 AM whitequark: oh I see
09:54 AM whitequark: right so the idea with Glasgow is that it takes care of all the annoying parts of FPGA toolchains (by cutting very carefully measured corners).
09:54 AM pcw_home: all these things are possible but you sort of end up re-implementing linuxCNC/HAL on the remote device
09:54 AM whitequark: one aspect of it is that something like torch height control (assuming you have some kind of ADC for that) is maybe a few added lines of code
09:54 AM pcw_home: its not...
09:54 AM pcw_home: THC is complex
09:55 AM whitequark: well, you have a transfer function with feedback. if you can define it in integer domain, it's not hard to add
09:56 AM whitequark: it's no point arguing, I'd much rather write a PoC for complex feedback on FPGA. worth a thousand words.
09:56 AM pcw_home: THCs are much more complex (corner locking, kerf crossing etc etc)
09:56 AM whitequark: ohh I see
09:56 AM whitequark: that's interesting then.
09:56 AM pcw_home: hal, unlimited floating point make this easier to do
09:58 AM pcw_home: there is a very large set of linuxCNC uses that dont need these features (routers etc)
09:58 AM whitequark: new plan: instead of replacing HAL, extend HAL into the FPGA, by making it possible to define HAL signals across the USB interface.
09:58 AM rmu: rt-preempt kernels are not that problematic if you have real hardware
09:59 AM pcw_home: yes, I suspect that (remote hal) is a better choice (unless you only aim at simple targets)
09:59 AM whitequark: I assume this plan would need rt-preempt anyway, since HAL is based on rt
10:00 AM whitequark: remote HAL, hmm.
10:00 AM whitequark: it's kind of annoying here that HAL is polling driven
10:01 AM whitequark: logically, I have a stream of tuples with every HAL parameter used, which on each cycle are compressesd (by differentiating against previous tuple), and queued into the device. then the same on the other side for inputs.
10:02 AM rmu: are you sure compression is worth it? USB should not have a bandwidth problem, there is not that much data to transfer
10:02 AM whitequark: 2 points on this.
10:03 AM whitequark: first, it's not real compression, more just not sending the data you don't need. if nothing else, this helps with debugging.
10:03 AM whitequark: second, you'll be surprised how many problems USB has.
10:03 AM whitequark: honestly, it is mostly problems.
10:04 AM rmu: that's the reason nobody did a linuxcnc usb interface yet
10:04 AM whitequark: the bounded latency transfer mode it has doesn't have guaranteed delivery, and the guaranteed delivery mode doesn't have bounded latency.
10:04 AM whitequark: this is -in spite of- the bus and host controllers allowing reserving chunks of bandwidth, -and- the bus inherently having no conflicts as it is driven by host using timeslots.
10:05 AM rmu: look at hostmot2 and pico servo interfaces for polled vs. interrupt transfers. both possible with hal.
10:05 AM whitequark: USB is like if someone had a box with bus LEGOs and decided to make the worst possible bus out of that.
10:05 AM whitequark: you trade off so many useful potential features, and receive a negative return on it.
10:05 AM rmu: but it's cheap
10:06 AM rmu: no firewire-patent-tax
10:06 AM whitequark: but it's so utterly, pointlessly horrible on every level.
10:06 AM whitequark: why in the actual hell does it use NRZI with bit stuffing?
10:06 AM whitequark: was 8b10b too hard?!
10:06 AM whitequark: now your physical packet length depends on the data inside the packet.
10:07 AM whitequark: you can't even decode a packet in constant time unless you know what's inside!
10:07 AM whitequark: and I mean, sure, maybe it saved someone five hundredths of a cent per chip in 1993, but there's no excuse to keep that in USB 2.
10:07 AM whitequark: it's a differential bus, but it's *also* dc coupled.
10:07 AM rmu: use ethernet then
10:08 AM whitequark: moreover, it has differential states interacting with dc coupled states, which means you can't even use a differential transceiver to implement it.
10:08 AM whitequark: you *have* to implement it as two single ended transceivers, with a sort of differential kludge on the side.
10:09 AM whitequark: I could go on for weeks.
10:09 AM whitequark: anyway.
10:10 AM whitequark: Ethernet is good and all but it doesn't improve things a lot.
10:11 AM rmu: somehow reminds me of https://youtu.be/Br2fsbH9sy8?t=13
10:11 AM whitequark: Ethernet doesn't have guaranteed delivery unless you build it yourself, and the latency, instead of depending on your utterly broken OS driver stack, also depends on your utterly broken network hardware.
10:11 AM whitequark: I bet if I used Ethernet, someone would try to use remote HAL over public WiFi, and complain.
10:11 AM rmu: just use 1:1 connection to dedicated network card
10:12 AM whitequark: I have a laptop that has three USB ports and that's it.
10:12 AM rmu: usb ethernet dongle then :P
10:12 AM whitequark: (ok, one of them is technically Thunderbolt, but Thunderbolt is also batshit insane and I'll leave it as that. I am unfortunately very well aware of it.)
10:12 AM whitequark: that doesn't work.
10:13 AM whitequark: that gives you all the problems of USB *and* Ethernet.
10:13 AM rmu: it was a joke
10:13 AM rmu: there is no perfect solution
10:13 AM whitequark: I would have used Ethernet if all computers still had dedicated Ethernet ports, but the world has sadly moved on to really thin laptops.
10:13 AM whitequark: which I also really like.
10:13 AM rmu: simple, cheap, sane, correct. chose 1.
10:14 AM whitequark: if they made an USB type C SGMII alternative mode, we *would* have a sane solution to this that works perfectly well.
10:14 AM whitequark: I've dug out the proposal someone made for it.
10:15 AM whitequark: no one involved with USB type C was interested, which is unsurprising, because everyone involved with USB-C ranges from terminally incompetent to shouldn't go near a single electronic device much less a sheet of paper with "specification" written on it.
10:15 AM pcw_home: I ran one of our Ethernet cards on a USB Ethernet dongle, it worked ok for maybe an hour until it tripped over its shoelaces...
10:15 AM whitequark: I could tell you *so much* about the absolute worst aspects of USB and Thunderbolt.
10:16 AM whitequark: the only way Intel was able to keep Thunderbolt from falling apart is by mandating that every certified TBT3 device follow the exact schematics of one of the ~five Intel reference designs. down to individual BOM lines.
10:16 AM whitequark: if you don't do that, they sue you off the market with their patent pool.
10:16 AM whitequark: that should tell you everything you need to know about Thunderbolt.
10:17 AM whitequark: (also, when you plug a TBT device in, for it to work, you need USB PD--that's power delivery, EC, ACPI, SMM, and OS to cooperate. at *least* six different firmwares are involved.)
10:17 AM rmu: for a typical machine, you would need to transfer around 100 bytes max in either direction with 1kHz update rate, and if something bad happens for long enough you "just" have to E-Stop from the FPGA side.
10:18 AM whitequark: right, so bandwidth is not really a problem
10:18 AM whitequark: the thing about USB is that it's based on timeslots, and it actually doesn't matter a lot *how* much data you transfer.
10:18 AM whitequark: what does matter, is in how many packets, and so timeslots, it will end up being split.
10:19 AM whitequark: on certain configurations, I could take up most of the bus bandwidth with 1000 bytes per second.
10:19 AM rmu: if you generate steps in the FPGA jitter of this 1khz update will not be that critical and missing some updates usually is not catastrophic
10:19 AM whitequark: well, enough to make it unusable.
10:19 AM pcw_home: even for more bandwidth heavy uses (laser painting) its still pretty small
10:20 AM whitequark: the problem with USB isn't so much that it has not enough bandwidth (more than enough) or that it doesn't guarantee latency (it's usually pretty good), it's that it has a lot of strange behavior that makes it work very much not like a FIFO
10:20 AM pcw_home: yeah if you assume you motion engine is running in velocity mode, rather ragged updates are ok
10:21 AM whitequark: question.
10:21 AM pcw_home: as long as you send TP timestamps...
10:22 AM whitequark: what's better, low but unbounded average latency with high jitter, or low and bounded latency for almost all requests and even higher spikes in rare circumstances?
10:23 AM whitequark: you always have to account for the case where you send an update and it never arrives, but you can trade it off a bit.
10:23 AM whitequark: former: plain bulk requests. latter: isochronous requests with a framing on top of them that ensures guaranteed delivery.
10:25 AM whitequark: what you *want* is bulk requests with a chunk of bus bandwidth explicitly reserved for them, and it's even possible to implement in a hacky sort of way, but no OS will actually let you do that, because that'd actually make USB useful.
10:25 AM whitequark: so you have to choose between sharing bandwidth with a mouse, and handling packet loss in software.
10:26 AM rmu: i don't know. assuming usb soundcards use isochronous transfers i would use that, never seem to have problems with that (no glitches to hear)
10:26 AM whitequark: there are two problems with isochronous transfers
10:26 AM rmu: but it is reasonable to assume that the CNC machine is on one dedicated USB bus
10:26 AM whitequark: first, they're buggy as hell in every OS and every device. that USB Audio works at all is nothing short of a miracle, and almost every complex configuration doesn't work well.
10:27 AM whitequark: e.g. there are USB sound cards that only work on specific USB host controllers.
10:27 AM pcw_home: Preempt-RT hosts are much like the latter: http://freeby.mesanet.com/8300-4.18.png
10:27 AM whitequark: second, isochronous transfers don't even *try* to acknowledge the transfer.
10:28 AM rmu: complex configuration don't work because of other aspects, not the transfers i assume
10:28 AM whitequark: the host controller *could* in theory be able to tell you if the packet went nowhere in reality, but it's not in the spec, so even though that information would have been trivial to request, it's impossible.
10:28 AM whitequark: no, the transfers.
10:28 AM whitequark: all the host controller does is schedules transfer from a linked list of DMA buffers
10:28 AM whitequark: you'd think that's hard to screw up, but wait until you see USB HCs.
10:29 AM whitequark: then the OS adds a few layers of crap on top of it and well
10:29 AM whitequark: libusb on Windows is like 1/5 as fast as libusb on Linux and glitches out constantly, and as far as I can tell that's not really fixable for the most part
10:29 AM pcw_home: Often in RT stuff, dropped data is simple ignored, where its important (things that change remote state) higher level handshaking should be used
10:30 AM whitequark: yep, so if I went for isochronous and a generic remote HAL mechanism, I would need to implement this handshaking on top of USB.
10:31 AM whitequark: I'd optimistically send all data as it comes to my remote HAL implementation, and then if it turns out there's a missed packet, I have to resend all of the buffered data.
10:31 AM rmu: or just ignore the missed update
10:32 AM whitequark: this, of course, is a problem, because USB isochronous transfers aren't elastic. you can't opportunistically give the bus a bit more data this cycle, in hope that it'll fit, you have to pre-reserve all of the bandwidth, and all of it will actually get used.
10:32 AM rmu: so you would need some kind of framing mechanism so you FPGA side recognizes begin of an update frame regardless of possible garbage and drops in between
10:32 AM whitequark: even worse, you can't even ask the host how much free bandwidth it has.
10:32 AM pcw_home: even there, there are only a couple of critical things that are not constantly re-sent
10:32 AM rmu: and some kind of watchdog that E-Stops the whole shebang if no update is receive within 100ms or so
10:33 AM whitequark: all real devices export a zillion isochronous interfaces and then enumerate between them until the HC/OS finally considers one acceptable.
10:33 AM whitequark: of course, when this breaks, it's impossible to debug, too
10:33 AM whitequark: hmm.
10:34 AM whitequark: I imagine it'd still be cleaner to use an isochronous endpoint...
10:34 AM whitequark: 16:29 < rmu> so you would need some kind of framing mechanism so you FPGA side recognizes begin of an update frame
10:35 AM whitequark: you don't need framing, because USB packet drops will never cause your boundaries to shift arbitrarily
10:35 AM whitequark: 16:29 < pcw_home> even there, there are only a couple of critical things that are not constantly re-sent
10:35 AM whitequark: but, you need some mechanism to separate the HAL data that can be dropped, vs HAL data that will need to be sent again.
10:36 AM rmu: if your motion control is in the FPGA it should not be that critical if you miss one update from the PC.
10:37 AM whitequark: if this is an update to something like "turn coolant on" it better arrive at some point
10:37 AM whitequark: hmm, I can see why the compression scheme I described above would be problematic.
10:37 AM rmu: you transfer the complete input/output process data each update cycle
10:37 AM whitequark: if *every* frame would include *every* HAL wire, I wouldn't need that
10:37 AM whitequark: yeah
10:38 AM pcw_home: That really should be done in LinuxCNC AFAICR the only critical (non duplicated) data sent is the mode setting for index encoder index detection or setup information
10:38 AM whitequark: that would simplify everything considerably, and actually make using isochronous fairly straightforward.
10:39 AM whitequark: rmu: thanks for this input, I will now design this thing in a completely different way.
10:39 AM rmu: my "big" cnc machine has about 80 digital inputs and 120 digital outputs, 4 axis (that's a couple of float per axis) and thats it.
10:39 AM whitequark: than I originally intended.
10:39 AM rmu: so not much to transfer really
10:39 AM whitequark: yes, it easily fits into one isochronous packet... let me look it up...
10:40 AM rmu: timestamp everything, and just keep sending. anything missed -> ignore.
10:40 AM rmu: looking forward to some updates ;)
10:40 AM whitequark: right, so the *minimum* meaningful isochronous packet size is 1024 bytes.
10:41 AM whitequark: if you have more than 1K of HAL parameters that's something very unusual.
10:41 AM rmu: that's probably an extreme edge case you can safely ignore
10:41 AM whitequark: I think I can get the latency down to 1/2 ms. definitely under 1 ms.
10:42 AM whitequark: not less than 1/4 ms under any circumstances, for a roundtrip.
10:43 AM rmu: you can decouple reading of inputs from your transfers
10:44 AM rmu: (snapshot at a specific time, synchronise via PLL with host)
10:45 AM whitequark: can you elaborate on that?
10:46 AM whitequark: the PLL is not needed, as the latency of output-to-input loop (for example) is constant and depends only on the FPGA gateware, provided the gateware is done correctly
10:46 AM whitequark: but I'm curious about the synchronization you propose.
10:46 AM rmu: assume you have a link that has some jitter, then it can make sense to trigger a "snapshot" of inputs independently, running with some exact clock
10:47 AM whitequark: ahh, I see what you mean.
10:47 AM rmu: so your inputs are always sampled at a precise interval
10:47 AM pcw_home: We do this on our Ethernet cards (for example when LinuxCNC is running the PID loop, and we want jitter free sampling of the encoder position)
10:48 AM whitequark: yes, that would of course be useful. the input sampling would be done at exactly the isochronous packet rate.
10:48 AM pcw_home: well jitter reduced...
10:48 AM whitequark: the FPGA is normally running synchronous to USB clock, so that's fairly easy.
10:49 AM pcw_home: this works pretty well because of the jitter statistics
10:51 AM whitequark: I'm currently thinking of taking my 3D printer, throwing out the Arduino junk it came with, and replacing everything from the G-code parser down to and including FET drivers with my FPGA board.
10:51 AM whitequark: do the chopping and dead time generation in gateware too, why not.
10:58 AM whitequark: hmm, so a fun thing to do would be to make the USB device self-describing.
10:59 AM whitequark: you load a HAL module, it finds the device, pulls the HAL wire definitions out of it.
12:39 PM whitequark: is there an intro to rt-preempt that you would recommend?
12:41 PM rmu: I only know the linuxfoundation wiki https://wiki.linuxfoundation.org/realtime/documentation/howto/applications/application_base
12:42 PM rmu: but building a hal module is pretty easy, then the rt stuff is taken care of
12:47 PM whitequark: i'm wondering about the use of usb from realtime domain
12:47 PM whitequark: i mean, the whole thing is a hal module
12:48 PM whitequark: if it has any other component in linuxcnc, it'd just request the wire format via usb
12:48 PM whitequark: hang on
12:49 PM whitequark: rmu: if i'm using rt-preempt is there any benefit to doing the hal component as a kernel module as opposed to an userspace process?
12:59 PM pcw_home: I dont think so (all normal hal RT modules are userland AFAIK unless you are using RTAI)
12:59 PM whitequark: oh well, that simplifies my job massively
01:00 PM pcw_home: also makes them much easier to debug
01:01 PM whitequark: >In a Run-In-Place build, the LinuxCNC programs are compiled from source and then run directly from within the build directory. Nothing is installed outside the build directory.
01:01 PM whitequark: I love this
01:01 PM whitequark: I was dreading installing something to /usr/local
01:01 PM whitequark: I think that step alone made me like linuxcnc twice as much
01:14 PM Tom_L: pretty soon you may wanna just use it as it is
01:15 PM whitequark: hm, dpkg-checkbuilddeps is satisfied, but configure still fails
01:15 PM whitequark: checking for yapps... none
01:16 PM whitequark: i have a yapps2 binary
01:18 PM whitequark: i also had to install bwidget, libtk-img and tclx. missing dependency?
02:24 PM sliptonic: Hey everyone, I'm working on a 4th axis surfacing operation for FreeCAD. I'm generating gcode that looks good but when I run it, linuxcnc stops on a particular line with no error. It just stops moving. How can I figure out what's going on?
02:34 PM rmu: sliptonic: any message in the console?
02:38 PM sliptonic: No. I seen an Emit Interp-run when I start it, but nothing when it stops
02:39 PM sliptonic: Linuxcnc/gmoccapy is not frozen. I can stop the job, restart, jog, MDI, everything.
02:39 PM sliptonic: It just refuses to run past the line.
02:40 PM hazzy-m: sliptonic: what state does it stop in? READING, WAITING, IDLE?
02:41 PM sliptonic: I'm not sure. How do I determine state?
02:48 PM hazzy-m: sliptonic: you can use the LinucCNC Status tool (available from the Machine menu in Axis, and I think from the settings page in Gmoccapy), or from the command line as linuxcnctop
02:49 PM hazzy-m: watch the interp_state value, that might give you a hint were it is hanging
02:49 PM hazzy-m: you can also look at motion_type, to see what motion it hangs on
02:50 PM skunkworks: pcw_home: the 'WARNING: this firmware has Muxed Encoder v2!' should be fixed after a apt-get update/upgrade?
02:57 PM skunkworks: hmm - no.
03:01 PM skunkworks: This is for 5i20...
03:02 PM skunkworks: I just installed stretch on the K&T... Went very smooth..
03:02 PM skunkworks: (and I have a special comp and arduino interface...)
03:05 PM pcw_home: needs newer firmware, probably fixed if you fetch bitfile from Mesa
03:06 PM pcw_home: that's an _old_ bug
03:06 PM pcw_home: maybe 10 years or more
03:07 PM skunkworks: heh
03:08 PM * Tom_L feels skunkworks should be running state of the art firmware
03:08 PM skunkworks: I figured. So just get the bit files from mesa and put them (where do they go?)
03:09 PM pcw_home: "/lib/firmware/hm2/5i20"
03:10 PM hazzy-m: skunkworks: what arduino to HAL interface do you use?
03:11 PM skunkworks: sound it
03:12 PM skunkworks: found it
03:12 PM pcw_home: firmware bug is incorrectly scaled velocity estimate with muxed encoders (a copy/paste error)
03:12 PM skunkworks: hazzy-m: https://emergent.unpythonic.net/01198594294
03:12 PM skunkworks: non-realtime but it is just for temp compinsation
03:13 PM hazzy-m: skunkworks: Thanks! That is the same one I use
03:13 PM hazzy-m: there is also a C based comp one
03:14 PM hazzy-m: skunkworks: I cleaned jeff's up a bit and made it installable with pip: https://github.com/KurtJacobson/arduino-hal-interface
03:14 PM sliptonic: hazzy-m: interp_state is 2 and motion.motion_type is 1
03:16 PM hazzy-m: you can decode those here: https://qtpyvcp.kcjengr.com/components/status_items.html#interp-state
03:17 PM hazzy-m: so it looks like it hangs while reading ...
03:17 PM skunkworks: cool!
03:18 PM hazzy-m: usually it finishes reading almost immediately after starting the program, so that seems a little odd
03:18 PM skunkworks: pcw_home: I downloaded the support software and the SVST8_4.BIT is dated 09-sep-11
03:19 PM skunkworks: pcw_home: the on in firmware is dated 2015
03:21 PM * hazzy-m uploaded an image: 1551647875778.jpg (288KB) < https://matrix.org/_matrix/media/v1/download/matrix.org/HSEbitaSGBErVmDmVdeGpkIW >
03:21 PM hazzy-m: oops, sorry wrong chan
03:22 PM sliptonic: I think it's unrelated to my program. Looks like the spindle-at-speed detection is having trouble.
03:22 PM sliptonic: Thanks for the help
03:23 PM hazzy-m: sliptonic: Ah, good to hear
03:28 PM skunkworks: actually - it is the SV12_2X7i48_72.bit
03:28 PM skunkworks: but I don't see it in the zip from mesa
03:43 PM pcw_mesa: pretty sure you can use the"IM" versions
03:58 PM -!- #linuxcnc-devel mode set to +v by ChanServ
04:28 PM skunkworks: pcw_home: ok - I will try it when I am back in the shop