#linuxcnc-devel Logs

Oct 12 2023

05:44 AM rigid: rene-dev5: i'm not so sure since you keep asking the same question three times and don't pick up any of the arguments in my answers. As explained, I do not claim NML enabled tooldata is necessary for a remote UI. I'm claiming, the original linuxcnc design allows running the tooldb remotely.
05:53 AM rigid: So instead of repeating myself, let me instead ask how you think people networked/centralized their tooling data in previous releases of linuxcnc other than plugging into the CMS to integrate their solution frictionless into the realtime system?
05:56 AM rmu: the realtime stuff doesn't need access to tooltable AFAIK
05:57 AM rmu: and regarding databases, sqlite is one of the really robust and fast ones
05:58 AM rmu: esp for simple stuff (single table, no joins)
06:00 AM rigid: Hm, so either toolchangers can't use HAL or they are not tracked in the DB. It's possible that I still misunderstand the architecture but I can't imagine the tooldb plugging into a realtime component at some place, yet.
06:01 AM rigid: sqlite is far from fast. Basically any other SQL is faster and in terms of speed/latency, stuff like redis or nosql was invented.
06:01 AM rmu: every other database involves a round trip into another process, that can't be faster
06:02 AM rmu: the tooldb isn't doing anything fancy, it doesn't even need an index, linear scan is no problem for 1000s of lines, and who hast that much tools in the tool db
06:03 AM rigid: yeah, because you want another process. That's not just for seperation-of-concerns. SQL does logging/journaling which can lock up your task at any time for an unpredictable duration.
06:03 AM rmu: remapped M6 isn't running in realtime
06:05 AM rmu: sqlite as it will be used in tooldb is just C-struct on steroids, easily extended with additional fields if you want to track tool life, RFIDs or max spindle speed
06:05 AM rigid: rmu: that's the other point: People do all sorts of fancy stuff with their tooldb. Why remove those capabilities?
06:06 AM rmu: don't understand what capability is going to be removed?
06:06 AM rigid: for example: will [EMCIO]DB_PROGRAM stay?
06:07 AM rigid: or toolSts and toolCmd CMS?
06:08 AM rigid: (which only ever partially worked afais)
06:13 AM rmu: removing stuff that didn't work isn't removing capability
06:15 AM rigid: you mean "don't work right now". [EMCIO]DB_PROGRAM was added to solve a problem and surely worked at some point. People are running _ancient_ versions of LinuxCNC
06:16 AM rigid: i'd rather fix stuff than just ripping out code that seems to be broken
06:16 AM rigid: and even more with core components like NML
06:16 AM rigid: ...without any RFC or writeup
06:16 AM rigid: ...in master
06:18 AM rigid: (btw, it's just right now where people care about updating production firmware since ppl slowly realize that network security is actually a thing :-)
06:21 AM rmu: i have no idea if tool_db is actually working or not, as grep DB_PROGRAM in tests/ comes up empty I would assume if it does work it is by accident
06:22 AM rmu: NML surely is a beacon of how to implement secure networking protocol
06:22 AM rigid: i didn't test it either but configs/sim/axis/db_demo still exists
06:24 AM rmu: i do know that tool table in EMC_TOOL_STAT message can cause problems because of it's size
06:24 AM rigid: it seems to be an axis thing. it'd be awesome if all UIs could use that transparently via one single API.
06:25 AM rigid: yeah, this has been mentioned before as well as a 50-tool limit. I can't reproduce that.
06:26 AM rmu: a 1000 tool table is possible, no problem, but IIRC with state tags, this status structure gets a copy for every g-code "in flight", and that can quickly multiply
06:27 AM rmu: combined with RTAI kernel realtime system, that can be a real problem, kernel memory is not pageable
06:27 AM rigid: ah I see. I think that's why there are different modes for status updates
06:28 AM rigid: isn't RTAI depreciated for uspace?
06:28 AM rmu: tell that to the guys running software step generation on parallel ports
06:29 AM rigid: most of those probably use rt patches
06:29 AM rmu: ?
06:29 AM rigid: I didn't even try RTAI since I didn't use it in years
06:32 AM rigid: rmu: you can either compile linuxcnc for RTAI or for RTLinux patches (uspace) (those are two different ways to make the kernel realtime capable).
06:33 AM rigid: now I'm confused but iirc, RTAI is pretty much depreciated by RTLinux, linuxcnc doesn't use it by default. Please correct me.
06:35 AM rigid: and yeah, you want to have proper lock/timeout handling for your tooldb without affecting the realtime components. sqlite will also impose those problems at some point.
06:36 AM rigid: also you might want to run a huge DB cluster with thousands of tools and a plethora of properties.
06:38 AM rene-dev5: rigid I dont think you fully understand the architecture of linuxcnc, or how the toolchange works.
06:38 AM rigid: like no one goes around inspecting a big production and goes like "this tool should be replaced. better order a new one soon." It's automated to scale.
06:39 AM rigid: Good. I mentioned it multiple times that I might miss things. Very good.
06:40 AM rene-dev5: nothing about the tooltable is realtime
06:40 AM rene-dev5: there is no ipc. only task and the UI access the tooltable.
06:41 AM rigid: so what mru said "EMC_TOOL_STAT message can cause problems ... combined with RTAI kernel realtime system, that can be a real problem" isn't true?
06:41 AM rene-dev5: unlike a file, you can access a sqlite db from multiple processes.
06:41 AM rigid: rene-dev5: [EMCIO]DB_PROGRAM is not IPC? or toolSts? How is that not IPC
06:42 AM rigid: the .tbl file is just the backend, abstracted by tooldata.hh API. You could not only access it from multiple processes but also from different hosts using the .nml config
06:42 AM rene-dev5: sqlite is really fast for reading.
06:43 AM rigid: and? a text file is fast aswell
06:43 AM rigid: or a postgresql cluster
06:43 AM rene-dev5: the DB_PROGRAM stuff isnt used by the UI, so how is it useful in any way?
06:43 AM rigid: the DB_PROGRAM stuff is used by axis
06:44 AM rene-dev5: a text file cant be accessed by multiple programs at the same time. you get a race condition.
06:44 AM rene-dev5: linuxcnc currently does that.
06:44 AM rigid: genereally: if you have to ask how it's useful, I wouldn't touch it without asking... in master
06:45 AM rigid: linuxcnc doesn't "do that". Some components do that since ppl didn't understand the system design and implemented stuff wrongly. There's no reason not to just fix that.
06:45 AM rene-dev5: the DB_PROGRAM stuff is really badly implemented.
06:46 AM rigid: and? it solved a problem. If you'd remove everything badly implemented from linuxcnc...
06:46 AM rigid: there'd be almost nothing left.
06:46 AM rigid: it's FOSS
06:47 AM rmu: rigid: AFAIK RTLinux is EOL, RTAI was a spin-off 20something years ago that did something different to avoid the RTLinux patents and is stull around. then there also is xenomai.
06:48 AM rigid: rmu: right, i mixed up RTLinux with rt-preempt
06:48 AM rigid: that's what's used by default in linuxcnc
06:48 AM rmu: rt preempt is something completely different
06:49 AM rigid: it's what pretty much depreciated RTAI in mainline as ubuntu started to provide official rt-preempt kernels
06:49 AM rmu: getting rid of kernel space real time systems is not on the agenda
06:49 AM rene-dev5: rt preempt is what most pepole use, supported by the mainline kernel, and the only rt system that works on modern hardware
06:50 AM rene-dev5: rigid ubuntu doesnt offer rt kernels, debian does.
06:51 AM rmu: https://github.com/NTULINUX/RTAI/releases
06:51 AM rigid: https://canonical.com/blog/real-time-ubuntu-is-now-generally-available
06:51 AM rmu: rigid: paywalled
06:52 AM rigid: i didn't say ubuntu runs the show. But when it started LTS support, RTAI became obsoleteish
06:52 AM rigid: rmu: canonical doesn't paywall their blog
06:52 AM rigid: works fine here
06:52 AM rmu: whatever, nobody cares what kernel ubuntu provides, it's easy enough to build
06:52 AM rene-dev5: ah, thats new to me. thanks for mentioning it.
06:52 AM rmu: the kernel is paywalled
06:53 AM rigid: yeah, I mixed rt-preempt and RTLinux which caused confusion
06:54 AM rmu: ubuntu rt preempt kernel is avaiable for ubuntu pro or whatever they call it customers
06:54 AM rene-dev5: I cant find it in the repo
06:55 AM rmu: rene-dev5: you need subscription to ubuntu pro
06:56 AM rene-dev5: rigid you said that axis uses the tooldb stuff. how and where?
06:58 AM rigid: rene-dev5: check configs/sim/axis/db_demo
06:58 AM rigid: mentioned it before. you probably missed it.
06:59 AM rene-dev5: https://github.com/LinuxCNC/linuxcnc/blob/master/configs/sim/axis/db_demo/db_editor.tcl
06:59 AM rene-dev5: it doesnt allow you to edit the tools.
07:00 AM rigid: you mean the demo isn't a fullblown implementation? could be.
07:00 AM rene-dev5: I consider editing tools a rather basic feature
07:00 AM rigid: editing tools might be something needed to be added to the API even. The author probably didn't need it.
07:01 AM rene-dev5: its next to impossible to add new fields like sister tool, next tool, lifetime, max rpm, feedrate,...
07:01 AM rigid: when you run a db, you have means to edit it. i wouldn't implement another editor in a component that wouldn't need it.
07:01 AM rigid: "next to impossible"? why?
07:02 AM rene-dev5: have you looked at the code? like I said, it doesnt solve *any* of the problems with the tooltable.
07:02 AM rigid: sure, it's undoubtly easier to remove everything and patch sqlite in. but that doesn't mean one should actually do it.
07:03 AM rigid: you mean the DB_PROGRAM code or toolCmd/toolSts?
07:03 AM rene-dev5: sqlite and an api that actually works.
07:03 AM rene-dev5: rigid well, I already did it.
07:04 AM rigid: yeah, but misses a ton of features and generally must be highly doubted to even be fit for purpose.
07:04 AM rigid: i know... and in master even
07:04 AM rigid: ...without proper testing before/after (hence the uninitialized value issue)
07:05 AM rene-dev5: you havent even seen the sqlite branch, as it isnt public.
07:05 AM rigid: i mean the tool stuff already in master
07:05 AM rigid: glad to hear there's a separate branch for sqlite
07:07 AM rene-dev5: could you find where the uninizialized value comes from? what makes you think its from a change I did?
07:08 AM rigid: rene-dev5: i'm not necessarily thinking this. implying that might be wrong. But the bug doesn't appear when compiling with --enable-toolnml
07:09 AM rigid: it might be dngarrett when he added the alternative to toolnml. But please don't take it personally. Blaming specific contributors doesn't help at all.
07:10 AM rigid: I mean, not breaking edge cases in master with drastic changes would be nice, but it's nothing uncommon or really serious in comparision.
07:12 AM rigid: i'm also just arguing about the general roadmap. There's nothing wrong with an sqlite db backend or even a sqlite branch that solves stuff until someone cleans up the rest of the code (if ever). But removing stuff without depreciation causes headaches and branches might diverge to a point, they will never be merged again.
07:12 AM rene-dev5: this is how the tooltable and toolchange currently works: https://www.youtube.com/watch?v=LFrdqQZ8FFc
07:13 AM rigid: I fear that linuxcnc ends up as a non-network desktop app although it was fully networked before. And that in times when the whole industry moves more and more to highly integrated networked environments.
07:13 AM rigid: lol
07:13 AM rigid: what you mean exactly? too many API layers?
07:15 AM rene-dev5: but having a proper db actually helps with network setup. it makes it easier.
07:15 AM rigid: I get the impression that you might think people added complexity for fun out of boredom since they "just didn't know better". Usually that can't be more wrong.
07:15 AM rene-dev5: yes, too many layers.
07:16 AM rigid: yeah, ODBC would solve that (or DB_PROGRAM) but how does one make sqlite networkable?
07:16 AM rene-dev5: its a common theme in the linuxcnc world.
07:16 AM rigid: yeah, the layers are there for a reason.
07:16 AM rigid: not saying it couldn't be cleaned up.
07:16 AM rigid: it IS messy, undoubtly
07:16 AM rigid: but it's not useless
07:17 AM rene-dev5: depends on the setup. either have an api to access the tooltable over whatever network protocol the UI uses(currently nml)
07:18 AM rene-dev5: I belive that you think that this is possible at the moment, but it is not.
07:18 AM rigid: nowadays people like to outsource those layers to 3rd party libs maintained by someone else. That way you keep functionality with removing complexity.
07:18 AM rene-dev5: or add anoter simple api to access the tooltable, seperate from task
07:18 AM rigid: but then the UI is your database manager?
07:19 AM rene-dev5: no, you can manipulate a db from different processes, thats the whole point
07:19 AM rigid: actually some linuxcncsvr instance is meant to be the database manager (DB_PROGRAM also was never meant to be part of axis according to the original EMC design)
07:20 AM rigid: processes. but not hosts. no one would use sqlite as a network db. tunneling through a UI to access it.
07:20 AM rigid: but yeah, it might be possible. You could also autohotkey an RDP session of DB Browser...
07:21 AM rigid: (which I've actually seen)
07:21 AM rene-dev5: actually you can use sqlite on a network share. but that would not be the prefered way to do it, but provide a proper api.
07:21 AM rene-dev5: this is another example. https://github.com/LinuxCNC/linuxcnc/pull/2531/files
07:21 AM rene-dev5: look at the amount of code required just for a single pin
07:22 AM rene-dev5: currently there isnt a generic and simple way of adding pins to linuxcnc
07:22 AM rigid: rene-dev5: as I said to rene, if you have a problem with verbose code and messy centralization while still keeping seperation of concerns: Don't use C or C++
07:23 AM rigid: rewrite that thing in rust
07:23 AM rigid: as said: it can be cleaned up a lot - but you never make it look like a modern language.
07:23 AM rene-dev5: you are welcome to do that, but linuxcnc happens to be written in c and c++
07:24 AM rene-dev5: that isnt true, I am also currently rewriting the python bindings, shrinking them from 2000+ lines to about 100 just by using modern c++
07:25 AM rmu: i wouldn't call linuxcnc "fully networked". even if you can NML over the network and you managed to somehow get it working, NML has a bunch of problems
07:25 AM rigid: yeah, and it's not even that bad. I've seen worse. It just got tons of contributions that were cutting corners of the original design which is quite understandable for FOSS code that doesn't have a fool-proof design documentation and/or examples for every single API.
07:25 AM rmu: it is basically unmaintained, there is no security, and no versioning of structures
07:26 AM rigid: rene-dev5: yeah and you could save more with something like swig. I didn't say C++ isn't better than C. But in rust you'd probably only need a few dozen lines maybe?
07:27 AM rigid: NML does have versioning
07:27 AM rigid: no one seems to care, tho. Tests still use 1.0
07:27 AM rigid: 2.something was the most recent reference I found
07:27 AM rmu: swig isn't going to save anything, quite the contrary
07:27 AM rene-dev5: rigid you are the only person I have met in 10+ years using linuxcnc that actually thinks nml is a good thing.
07:28 AM rigid: and security: fixing all those strncpy() compiler warnings would be a good start :) or not rolling your own telnet server like it's 1995 :-P
07:28 AM rene-dev5: the only good thing about nml is that it currently works.
07:28 AM rigid: well, I DO see it critically. But I totally see why it was invented in the first place. realtime IPC is hard.
07:28 AM rmu: swig is that inefficient at interfacing python <-> C++ that a separate wrapper generator was written for Qt
07:29 AM rigid: "almost". I found some bugs in libnml by just quickly looking over the code
07:29 AM rigid: but yeah, mostly
07:29 AM rmu: realtime IPC is just memcpy in shared memory
07:29 AM rigid: ah is it? never used it myself. but it saves writing a lot of code for sure.
07:30 AM rene-dev5: rigid nml isnt used for the realtime part.
07:30 AM rigid: you mean shm, not memcpy right? how do you care for concurrent access?
07:30 AM rmu: swig wrapper definitions are more code than pybind
07:30 AM rene-dev5: shm
07:31 AM rene-dev5: rigid task is the only writer, and motion is the only reader. there is a lock.
07:31 AM rmu: rigid: you don't
07:31 AM rigid: NML is used for the realtime part and SHM is just one way it supports buffer access
07:31 AM rene-dev5: nml is written in c++, c++ isnt used anywhere in the rt part for kernel reasons
07:32 AM rene-dev5: ist just a bunch of structs in shared memory.
07:32 AM rigid: rene-dev5: so locking. How do you prevent a lock not released for an unexpected long time? -> timeouts. Also queueing. Read/Write ACL... NML provides all that.
07:32 AM rmu: with the hal stuff in shared memory, you rely on atomicity of writes, so usually nothing bad will happen if the mesa driver is writing stuff out while userspace is currently manipulating the same bits
07:32 AM rigid: libnml is written in C++?
07:32 AM rmu: s/currently/concurrently/
07:32 AM rene-dev5: again, nml is c++ and c++ cant be easily used in the kernel.
07:33 AM rene-dev5: there is no locking for hal pin access. it is based on the assumption that double access is atomic. there is only a lock for modifying the structs.
07:35 AM rene-dev5: https://github.com/LinuxCNC/linuxcnc/pull/2479
07:35 AM rigid: rene-dev5: i don't see you say the libnml API isn't usable from C. And I don't understand why you care for kernel space. RTAI and rt-preempt both can run rt code in userspace?
07:35 AM rene-dev5: if you want to learn more about how task and motion communicate
07:36 AM rene-dev5: rtai runs entirely in the kernel. there are many things you cant do in the kernel.
07:37 AM rene-dev5: rt preempt runs in userspace, you can do anything in userspace.
07:37 AM rigid: and libnml provides an abstraction layer that currently implements "shmem", "locmem", "filemem", "phantom" and "globmem". I could easily add another way if I'd need it. And it's not libnml's fault if other components just patch into the quasi-standard shm without using the APIs meant for it.
07:39 AM rigid: rene-dev5: hm, only german wikipedia page says "One big advantage of RTAI is, that using the LXRT variant it's possible to run hard-realtime-tasks in userspace". English one doesn't mention it and I'm too lazy to search: https://de.wikipedia.org/wiki/RTAI
07:39 AM rene-dev5: motion <-> task doesnt use nm. https://github.com/LinuxCNC/linuxcnc/blob/fef0fa9292bb323695834ac184e60255c82b27f5/src/emc/motion/motion.h
07:39 AM rene-dev5: nml
07:40 AM rigid: but yeah, i have no opinion on using C++ realtime code or keeping it C. I welcome code cleanups and C++ can cleanup a lot.
07:41 AM rene-dev5: agree, but pepole want to keep rtai. My opinion is that its old, broken, and unusable. and it doesnt support ethernet hardware.
07:41 AM rigid: rene-dev5: how are those MOTION messages used then: https://linuxcnc.org/docs/html/code/nml-messages.html ?
07:42 AM rigid: yeah, i agree. it's probably ultra hard to even get a current RTAI kernel running. But it's a compile-time option so I don't really care.
07:43 AM rene-dev5: ui sends the nml message to task, and task sends it via shm(not nml) to motion
07:44 AM rene-dev5: task is not realtime. task is where the interpreter runs.
07:44 AM rene-dev5: only ui <-> task is using NML.
07:45 AM rigid: so that diagram is wrong: https://linuxcnc.org/docs/html/code/code-notes.html#_architecture_overview ?
07:45 AM rene-dev5: rtai keeps breaking, as there is currently no ci testing rtai
07:46 AM rene-dev5: yes, very wrong and also outdated.
07:46 AM rene-dev5: should be updated.
07:46 AM rene-dev5: says NML? so the autor probably wasnt sure.
07:47 AM rene-dev5: and io is gone.
07:47 AM rigid: ok, but there was a point where you could have one remote (non-realtime) task could control one or more motions
07:48 AM rmu: rigid: how does NML support versioning of messages?
07:48 AM rene-dev5: not that Im aware. motion and task use shm.
07:49 AM rigid: rmu: keys/sequence for specific messages, protocol version for message classes
07:50 AM rigid: rene-dev5: really no NML although they use the cms buffer structs? like emcStatus.motion ?
07:50 AM rigid: i'll have to dig deeper into the code
07:54 AM rigid: rene-dev5: afais motion uses emc/task/taskintf.cc to communicate with task and that includes emc_nml.hh https://github.com/LinuxCNC/linuxcnc/blob/master/src/emc/task/taskintf.cc#L27
07:56 AM rigid: ahh, and that uses usrmotWriteEmcmotCommand() from usrmotintf.cc which does it's own locking and status polling. lol, splendid.
07:56 AM rigid: https://github.com/LinuxCNC/linuxcnc/blob/master/src/emc/motion/usrmotintf.cc#L74
07:56 AM NoSpark: rene, if you quickley sketch out a better diagram on paper, snap a pic and send it to me, I can make a pretty version for the docs?
07:57 AM rigid: it emcmot_status_t even seems similarily constructed like a CMS object... hm
07:57 AM NoSpark: ideally with some links to parts of the code? I have looked through linuxcnc code a couple of times, but it is REALLY hard to get my head around
07:58 AM rmu: rigid: i see no way to add a field to e.g. EMC_JOINT_STAT_TYPE and keep everything backwards compatible
07:59 AM rigid: rmu: what kind of field would you want to add?
07:59 AM rigid: or do you mean EMC_JOINT_STAT_MSG ?
08:00 AM rmu: EMC_JOINT_STAT
08:00 AM rmu: and the _MSG
08:01 AM rmu: you would have to handle that explicitly
08:01 AM rigid: of course you need to handle version if you want to prevent breaking backwards compatibility. there's no way around that.
08:01 AM rmu: say i want to include some camera status
08:02 AM rigid: and keeping backwards compatible accross minor versions is also not strictly necessary
08:02 AM rmu: that would mean i have to rebuild every app that talks to it
08:03 AM rmu: with protobuf and other more modern approaches you can add fields sometimes even remove fields without breaking message compatibility
08:03 AM rigid: or you just add camera to NML, implement it in the one gui you use and enable other UIs to support any camera, since linuxcnc handles everything.
08:03 AM rigid: yeah, protobuf handles versioning for you but it doesn't mean stuff can't break.
08:04 AM rigid: i'm all for protobuf as long as it keeps networkable
08:05 AM rmu: yes of course, breaking stuff is easy
08:06 AM rigid: speaking of camera: i'm still trying to wrapping my head around how properly implementing that worklight would look like
08:12 AM rmu: what is there to implement?
08:13 AM rmu: just connect it to main power of machine :-)
08:15 AM rigid: yeah, fuck remote monitoring :-P
08:15 AM rmu: i put IP camera into machine
08:15 AM NoSpark: qt dragon has a camera function
08:16 AM rmu: work light can be controlled via motion I/O pins, you can enable that with M64 P<pin> / disable with M65 P<pin>
08:16 AM rigid: qt dragon needs qt
08:17 AM rigid: cameras need computer vision for proper remote monitoring...
08:17 AM NoSpark: definitely amnot suggesting adding QT to core linuxcnc
08:17 AM NoSpark: I pointed it out because it pipes the camera feed through opencv
08:17 AM rigid: it's like "oh machine no. 426 just turned on in the iowa plant. do this-and-that-now and note the time here-and-here"
08:18 AM rigid: NoSpark: ah
08:18 AM NoSpark: would you really want the cameras integrated into core linuxcnc?
08:18 AM rigid: no
08:19 AM rmu: rigid nobody and nothing prevents you from doing all the machine vision stuff you like
08:19 AM rigid: it'd be nice to have some component that can interface with all other components tho... maybe some sort of video<->linuxcnc interface to turn on/off cameras, change brightness, contrast, fps maybe?
08:19 AM NoSpark: It would be nice to have an easy method of edge finding
08:19 AM NoSpark: rigid: can't hal do that already with a couple of simple comps?
08:19 AM rigid: rmu: why would I use machine vision to monitor a machine light? it's a simple signal like a switch.
08:20 AM rmu: rigid: you tell me, i didn't bring up machine vision for proper remote monitoring
08:20 AM NoSpark: you can use machine vision to mechanically scan parts in, for setting origens and rotations
08:20 AM rigid: NoSpark: yes. but hal is only remotly usable via halrmt (which is a bit broken), not via NML.
08:21 AM rigid: NoSpark: integrating HAL so GUIs can use it via network using one common linuxcnc API is one of my goals.
08:22 AM rigid: yeah, people are also using cameras instead of limit switches
08:22 AM NoSpark: so, to be blunt, this isn't really a conversation about tool databases or camera APIs, but more like whether linuxcnc should start a move to a more RT IPC API, or continue with NML?
08:22 AM rigid: we're probably entering that era... but currently i'd stay with HAL :)
08:23 AM rigid: NoSpark: yes basically. I wouldn't totally subscribe to "RT IPC API" but more "uniform API and some network-capable IPC"
08:24 AM rigid: ...like NML (actually CMS) was once designed.
08:24 AM NoSpark: rigid: there are many use cases where CV is the only viable option, like PCB fiducial (bad spelling)?
08:26 AM rigid: yeah, totally. Even small manufacturers use X-Ray for checks. Although you could detect fiducial markings with simple photodiodes. iirc that's what they did in the past.
08:26 AM NoSpark: I'm just an outside observer to all this, so feel free to ask me (politely :) to stay out of this. is there a cost (time) benefit analysis that needs to be done?
08:26 AM rigid: but things certainly moved ones. they probably encode all sorts of stuff into QR codes now
08:28 AM rigid: NoSpark: that would be useful. but my main concern currently is that abandoning NML more and more removes features that need little time to use/implement/fix. Rarely used features but if you need them again in the future, a lot of time will have to be spent.
08:28 AM NoSpark: I'm trying to get to a point where I can be more involved with linuxcnc dev, but I'm a long way from there, as I almost exclusively work on embedded C systems, usually without linux at at.
08:28 AM rigid: i don't wanna know how many hours went into the "choose your locking mechanism" of NML alone...
08:28 AM NoSpark: Should someone make a list of actualy features that can't be ported?
08:29 AM NoSpark: keeping code around because it took a lot of effort to write is an example of the "sunk costs" fallacy.
08:30 AM NoSpark: especially if it is specific to linuxcnc and has to be maintained
08:30 AM rigid: hm, dunno. could be hard. it's like "what (future) features would be missing in app XY if you remove network access from your PC". It's either very short or very long
08:30 AM NoSpark: It is hard to argue for or against a change, if the costs of the change arn't known
08:31 AM rigid: I totally agree that there are cases where expensive code being removed improves the situation. But removing it gradually without a roadmap might cost more in the end than maintaining outdated, but somewhat working code.
08:31 AM rigid: yeah
08:32 AM rigid: but it's still a good thing to plan the future. I mean we all agree that NML needs to go sooner or later.
08:32 AM NoSpark: switching to an external library, written by experts specific to that library (libboost is an excellent example expertly created libraries) has hard, tangible benefits
08:33 AM rigid: yep... and there is more than one possible candidate that could replace NML + keep all/most of it's features
08:33 AM rmu: rigid: do you know https://github.com/DjangoReinhard/JCNCScreen
08:33 AM rigid: same btw. for linuxcncrsh. No one would roll her own readline nowadays. It's a perfect candidate for liblinenoise
08:33 AM NoSpark: if you know that it needs to go sooner or later, you should immediately assume ANY work on NML is wasted, unless it is bug fixes in previous versions of linuxcnc
08:34 AM rigid: rmu: no
08:34 AM rigid: rmu: looks nice. it also compensates for lack of tooltable abstraction in linuxcnc
08:34 AM rigid: would have been nice to build that into linuxcnc, not the GUI
08:35 AM rigid: maybe we could borrow some code
08:35 AM NoSpark: maybe conteact the dev and ask if he wants to port it in?
08:36 AM NoSpark: *contact
08:36 AM rigid: last commit was 2 years ago so... but IF this comes on the roadmap and IF someone starts working on it, i'd certainly notify the author and ask for participation
08:37 AM rigid: even if he'd just want to upgrade to adapt to a newly added tooldb capability in linuxcnc
08:37 AM rigid: *wants
08:37 AM rigid: maybe even a small and completely separate libtooldb would make sense
08:38 AM NoSpark: that reminds me, how hard would it be to break the motion planner into a library?
08:40 AM rigid: chance is answer this wrong is high. but at least using an external lib inside emc/motion should be doable. most effort probably would be keeping the old code around and write new tests.
08:40 AM rigid: *I
08:40 AM rmu: this JCNCScreen is using apache derby for tool storage
08:41 AM rigid: yeah, of course. nobody is perfect. glad it didn't use mssql or java
08:41 AM rmu: motion doesn't need tool db
08:41 AM rmu: derby is implemented entirely in java
08:41 AM rigid: derby is fullblown industry capable tho
08:41 AM rmu: yes very very enterprisey i'm sure
08:42 AM rmu: even comes with SQLConnectionListenerFactoryFactoryProviderFactory
08:44 AM rigid: I'm no DB expert but I still find ODBC the best solution for apps that don't need highly specifc DB features. But I didn't look further into DB_PROGRAM. It might be even more straight forward
08:44 AM rigid: but parsing some pipe input also seems totally out of fashion
08:45 AM rigid: (or cmdline arguments)
08:46 AM rigid: ODBC is a pain to setup for both user and dev, but once it runs, it "just works forever" (tm)
08:59 AM NoSpark: it seems like a LOT of projects keep reinventing the wheel with motion planners, and it seems like that effort could be spent improving the linuxcnc motion planner
08:59 AM rene-dev5: NoSpark https://drive.google.com/file/d/1oRseIPxa4Eo6xCbqn4QzcGzcBezUpqWS/view?usp=sharing
09:00 AM rene-dev5: quick sketch using draw.io
09:00 AM NoSpark: is axisui part of core linuxcnc?
09:00 AM rene-dev5: its in the repo
09:03 AM rene-dev5: more generic, I mean it could be any UI
09:03 AM rene-dev5: usually there is one
09:06 AM NoSpark: ahh, okay
09:11 AM rene-dev5: Do you want to clean that up, and make a pr?
09:11 AM NoSpark: I can
09:12 AM rene-dev5: would be great if every part references some file, so you know what is where
09:18 AM NoSpark: yeah, I expect I might have a few revisions ber everyone is happy with it
09:19 AM NoSpark: s/ber/everyone
09:19 AM NoSpark: s/ber/before
09:20 AM NoSpark: words are hard...
11:16 AM linuxcnc-build2: Build [#1690](http://buildbot2.highlab.com/buildbot/#builders/14/builds/1690) of `10-rip.debian-10-buster-rtpreempt-amd64` 4failed.
11:18 AM linuxcnc-build2: Build [#1691](http://buildbot2.highlab.com/buildbot/#builders/7/builds/1691) of `10-rip.debian-11-bullseye-rtpreempt-amd64` 4failed.
11:23 AM linuxcnc-build: build #3519 of 1660.rip-buster-python3 is complete: Failure [4failed compile runtests] Build details are at http://buildbot.linuxcnc.org/buildbot/builders/1660.rip-buster-python3/builds/3519 blamelist: Peter Wallace <pcw@mesanet.com>, Petter Reinholdtsen <pere@hungry.com>, andypugh <andy@bodgesoc.org>, Rene Hopf <renehopf@mac.com>, Phillip Carter
11:23 AM linuxcnc-build: <phillc54@users.noreply.github.com>, Norbert Schechner <nieson@web.de>, Sebastian Kuzminsky <seb@highlab.com>
11:24 AM linuxcnc-build2: Build [#1768](http://buildbot2.highlab.com/buildbot/#builders/13/builds/1768) of `00-checkin` 4failed.
11:33 AM linuxcnc-build: build #3921 of 1640.rip-buster-rtpreempt-amd64 is complete: Failure [4failed compile runtests] Build details are at http://buildbot.linuxcnc.org/buildbot/builders/1640.rip-buster-rtpreempt-amd64/builds/3921 blamelist: Peter Wallace <pcw@mesanet.com>, Petter Reinholdtsen <pere@hungry.com>, andypugh <andy@bodgesoc.org>, Rene Hopf <renehopf@mac.com>,
11:33 AM linuxcnc-build: Phillip Carter <phillc54@users.noreply.github.com>, Norbert Schechner <nieson@web.de>, Sebastian Kuzminsky <seb@highlab.com>
11:51 AM linuxcnc-build: build #10340 of 0000.checkin is complete: Failure [4failed] Build details are at http://buildbot.linuxcnc.org/buildbot/builders/0000.checkin/builds/10340 blamelist: Peter Wallace <pcw@mesanet.com>, Petter Reinholdtsen <pere@hungry.com>, andypugh <andy@bodgesoc.org>, Rene Hopf <renehopf@mac.com>, Phillip Carter <phillc54@users.noreply.github.com>,
11:51 AM linuxcnc-build: Norbert Schechner <nieson@web.de>, Sebastian Kuzminsky <seb@highlab.com>
01:45 PM rene-dev5: NoSpark did you get anywhere with the diagram?
01:58 PM pere: any hope to get the unstable tests in master to consistently succeed, or can they perhaps be disabled for now?
02:08 PM rene-dev5: which tests in particular?
02:12 PM rene-dev5: https://github.com/LinuxCNC/linuxcnc/issues/2644
02:12 PM rene-dev5: this one? looking at it now
02:23 PM pere: rene-dev5: none in particular, I just noticed several github build failures, and seb_kuzminsky told that the publishing of updated translations is stuck waiting for the tests to succeed.
02:26 PM rene-dev5: raster has a race condition where it tries to link the pin without waiting for the component to be loaded
02:27 PM rene-dev5: you could check that, except you cant, because raster doesnt use the normal python bindings, but brings its own bindings using ctypes
02:27 PM rene-dev5: https://www.youtube.com/watch?v=LFrdqQZ8FFc
03:25 PM rene-dev5: there are also a bunch of toolnfo tests failing
03:25 PM linuxcnc-build2: Build [#1698](http://buildbot2.highlab.com/buildbot/#builders/11/builds/1698) of `10-rip.debian-12-bookworm-rtpreempt-amd64` 4failed.
03:27 PM linuxcnc-build: build #3520 of 1660.rip-buster-python3 is complete: Failure [4failed compile runtests] Build details are at http://buildbot.linuxcnc.org/buildbot/builders/1660.rip-buster-python3/builds/3520 blamelist: Rene Hopf <renehopf@mac.com>
03:36 PM linuxcnc-build: build #3922 of 1640.rip-buster-rtpreempt-amd64 is complete: Failure [4failed compile runtests] Build details are at http://buildbot.linuxcnc.org/buildbot/builders/1640.rip-buster-rtpreempt-amd64/builds/3922 blamelist: Rene Hopf <renehopf@mac.com>
03:55 PM linuxcnc-build: build #10341 of 0000.checkin is complete: Failure [4failed] Build details are at http://buildbot.linuxcnc.org/buildbot/builders/0000.checkin/builds/10341 blamelist: Rene Hopf <renehopf@mac.com>
04:43 PM andypugh: I am trying to use Docker to build my local git folder under Buster. This is going poorly.
04:43 PM andypugh: Does anyone know anything about docker?
04:50 PM seb_kuzminsky: andypugh: i use docker to build linuxcnc and i'd love to talk with you about it, but i have to run - i'll be back in about 2 hours
04:51 PM seb_kuzminsky: if you send an email to emc-developers we can talk about it there maybe
04:52 PM andypugh: I think I will have either worked it out or given up entirely in 2 hours.
05:45 PM seb_kuzminsky: i'm back
05:45 PM seb_kuzminsky: andypugh: what problem are you having?
05:45 PM andypugh: All of them?
05:46 PM andypugh: I am trying to write a dockerfile. And to get that to build linuxcnc images
05:47 PM andypugh: I eventually found out that a dockerfile is not _allowed_ to ADD or mount or COPY anything from the local machine
05:47 PM andypugh: https://stackoverflow.com/questions/50585921/dockerfile-how-to-define-mounting-of-host-file-system-to-the-container
05:48 PM andypugh: So I am now trying to ADD the upstream git archive, which works to a limited extent. In that it creates _something_ in the container, but it’s not a directory.
05:50 PM andypugh: It took me ages to notice that VOLUME ~/linuxcnc-dev:/linuxcnc-dev was just creating a muntpoint called ~ containing a directory called linuxnc-dev:
06:19 PM seb_kuzminsky: i use a Dockerfile that installs what i hope is most of the build dependencies, then when i make a container from that image i bindmount my linuxcnc git repo into it, and do the normal thing of `debian/configure && apt build-dep .`
06:20 PM seb_kuzminsky: this way i can have a cached docker image that's 99% of what i need, and it's quick to install what's missing for some particular branch of linuxcnc
06:21 PM andypugh: I was hoping to write a script where I could choose the Linux version and the git tag and leave it to run.
06:22 PM seb_kuzminsky: that sounds useful
06:22 PM andypugh: Yeah, but probably not possible.
06:23 PM seb_kuzminsky: your script would have to generate the Dockerfile, since the FROM command at the top of the Dockerfile selects the base OS image to use
06:23 PM andypugh: As you can’t mount host directories in a dockerfile.
06:23 PM seb_kuzminsky: right, mounting of host directories happens on the `docker run` command line, not the `docker build`
06:24 PM andypugh: And there isn’t a “docker run” in a Dockerfile.
06:24 PM seb_kuzminsky: when you `docker run`, you can give it additional arguments which get executed inside the container, so you could place a script in the image which takes a tag name as an argument, and clones the git repo and checks out that commit, and runs the build
06:25 PM andypugh: The docs say that ADD can clone a git repo, but that seems not to work.
06:25 PM seb_kuzminsky: right, `docker build` creates the image (like a pbuilder chroot), and `docker run` creates a running container from that image (which is like starting the pbuilder chroot)
06:26 PM andypugh: It does something, and doenloads files, but doesn’t result in a source directory. (at least not for me)
06:26 PM seb_kuzminsky: use `RUN git clone $URL` if you want the git repo inside the image
06:26 PM seb_kuzminsky: (after `RUN apt install git` ofc)
06:26 PM andypugh: https://docs.docker.com/engine/reference/builder/#adding-a-git-repository-add-git-ref-dir
06:27 PM andypugh: “This form allows adding a git repository to an image directly, without using the git command inside the image:”
06:27 PM andypugh: Which I wasted an hour or so on.
06:27 PM andypugh: Choosing the Linux image is fairly easy, though.
06:28 PM andypugh: https://docs.docker.com/engine/reference/builder/#arg
06:29 PM seb_kuzminsky: right, something like `FROM debian:unstable` or `FROM debian:bookworm`
06:30 PM andypugh: ARG IMAGE="debian:buster"
06:30 PM andypugh: ARG LINUXCNC="~/linuxcnc-dev"
06:30 PM andypugh: FROM ${IMAGE}
06:30 PM andypugh: (the middle line isn’t relevant)
06:31 PM andypugh: docker build —build-arg IMAGE=debian:buster …
06:34 PM seb_kuzminsky: cool
06:38 PM andypugh: Well, it would be cool if the scripts could mount local files or public git archives, but one seems to be not possible (in the script) and the other seems to be not working.
06:41 PM seb_kuzminsky: after you `docker build ${WHATEVER}`, do `docker run --volume ${HOST_DIR}:${IN_CONTAINER} ls -la ${IN_CONTAINER}`
06:48 PM andypugh: I was hoping to have the debian build commands in the script too.
06:49 PM andypugh: I think to do what you say would require the script to exit fairly early?
07:28 PM seb_kuzminsky: no, i dont think so, the script can run as long as it wants and do whatever it wants inside the container
07:29 PM seb_kuzminsky: have you seen this? https://github.com/sebkuzminsky/linuxcnc-config-test