#linuxcnc Logs

Nov 07 2023

01:43 AM Deejay: moin
04:43 AM Tom_L: morning
04:44 AM Tom_L: 51F hi 70
04:50 AM JT-Cave: morning
04:53 AM Tom_L: 2.0.6 is ready
06:24 AM JT-Cave: got it
06:31 AM lcnc-relay: <TurBoss> morning
06:31 AM JT-Cave: rooster just crowed
06:41 AM Unterhaus_ is now known as unterhausen
12:23 PM CaptHindsight: https://www.deublin.com/en/Industries/Machine-Tool
12:56 PM unterhausen: this time hm2 only took two days to have an error finishing read. This seems bad. Is lengthening the thread interval going to help?
12:59 PM pcw-home: It most likely will though of course it depends on the latency statistics
12:59 PM pcw-home: what hardware do you have?
01:00 PM unterhausen: it's an i5 I think
01:01 PM unterhausen: dell optiplex, latency is pretty good
01:02 PM unterhausen: 35-ish us
01:02 PM pcw-home: Ethernet latency is usually the issue
01:02 PM pcw-home: what kernel?
01:03 PM unterhausen: I built it
01:03 PM unterhausen: 6.4
01:03 PM pcw-home: Do you have intel MACs?
01:04 PM unterhausen: yes
01:04 PM pcw-home: do you have irq coalescing disabled?
01:05 PM unterhausen: yes
01:06 PM pcw-home: Not sure then, I have had good luck with 6.3+ on intel hardware of that vintage
01:06 PM pcw-home: sure all power management C1 states >C1 etc are disabled?
01:06 PM -!- #linuxcnc mode set to +v by ChanServ
01:07 PM pcw-home: Are you using isolcpus?
01:09 PM pcw-home: You can also get an earlier warning of issue by checking hm2_card.0.packet-error-total
01:10 PM unterhausen: I'm pretty sure I'm using isolcpus
01:10 PM pcw-home: that is, you need to get a series of timeout errors in a row before you get a "error finishing red"
01:10 PM unterhausen: isolcpus = 2,3
01:11 PM pcw-home: "error finishing read"
01:12 PM pcw-home: I think I have got my best performance with isolcpus 3 and pinning the Ethernet IRQ to CPU 3
01:12 PM unterhausen: I tried changing the IRQ, but it's fixed
01:13 PM pcw-home: Not changing the IRQ but pinning it to the RT CPU (last CPU)
01:13 PM unterhausen: tjat
01:13 PM unterhausen: yeah, changing cpu affinity of the irq
01:14 PM unterhausen: maybe I should switch ports
01:14 PM pcw-home: you might try isolcpus 3 and setting the smp_affinity to "8"
01:15 PM unterhausen: unless I misunderstood, it's hooked up to the apic and can't change affinity
01:15 PM pcw-home: also if that's a corporate desktop disable ME/AMT
01:17 PM pcw-home: cd to /proc/irq/# and echo "8" > smp_affinity
01:17 PM pcw-home: (as root)
01:18 PM unterhausen: what does that do?
01:18 PM pcw-home: where # is the Ethernet IRQ
01:18 PM pcw-home: pins the Ethernet IRQ to processor 3
01:18 PM pcw-home: (this is assuming isolcpus 3)
01:20 PM unterhausen: when I tried that, it didn't do anything. I probably should try the other port, but it's a realtek
01:21 PM pcw-home: 8 because smp_affinity is a bit mask
01:22 PM pcw-home: It definitely affects Ethernet latency
01:22 PM unterhausen: my recollection is garbled, I'll check affinity again
01:23 PM pcw-home: it reduces the peak Ethernet latency on 3 or so test systems I've tried by about 30%
01:24 PM pcw-home: Watch hm2_card.0.read.tmax and hm2_card.0.packet-error-total to see how things are going
01:25 PM pcw-home: (to avoid waiting for days to see changes)
01:26 PM pcw-home: you can also try sudo chrt 99 ping -i .001 -q [card_ip_addr]
01:27 PM pcw-home: run it for a second or so then run again for 20 minutes or so
01:27 PM pcw-home: (first time to load ARP cache)
01:29 PM Unterhaus_: so I had it backwards the ethernet cards are not hooked up to an apic, so my understanding is I can't change affinity
01:31 PM Unterhaus_: IR-PCI-MISI or MSIX instead
01:31 PM pcw-home: I don't think smp_affinity has anything to do with the APIC
01:32 PM pcw-home: there's software between the actual hardware interrupt and actual scheduling
01:33 PM Unterhaus_: I was following this https://cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt
01:34 PM Unterhaus_: but them Canadians try to mislead occasionally
01:35 PM pcw-home: I think that's likely obsolete or not applicable to preempt-RT
01:38 PM Unterhaus_: affinity is set to 'f' right now
01:40 PM pcw-home: so all CPUs
01:40 PM pcw-home: setting it to "C" would make sense for isolcpus 2,3
01:41 PM Unterhaus_: servo period is 1 000 000
01:41 PM pcw-home: But, AFAIK linuxCNC will only use CPU 3
01:41 PM Unterhaus_: without the spaces
01:42 PM pcw-home: On my worst Intel CPU ( G800-G1 ) I have about 300 usec worst case Ethernet read time
01:43 PM pcw-home: after about a week of normal CPU usage (compilations, videos, FPGA firmware buiding)
01:45 PM Unterhaus_: maybe I should try a different kernel
01:45 PM pcw-home: Are you sure AMT is disabled?
01:45 PM Unterhaus_: I'm not sure how easy that would be on this system
01:46 PM pcw-home: its a BIOS setting
01:46 PM pcw-home: (well sometimes it is)
01:46 PM Unterhaus_: doesn't it show up regularly if it's a problem
01:46 PM Unterhaus_: this computer has an okay bios for a dell, but probably not that okay
01:47 PM pcw-home: yes but adds 100s of usec to peak latency
01:47 PM pcw-home: peak Ethernet lateny
01:47 PM Unterhaus_: but not cpu latency?
01:48 PM pcw-home: I just am trying 6.6-rt12 seems maybe a bit better than 6.4
01:48 PM pcw-home: No it does seem to affect CPU latency (which is really just scheduling latency)
01:50 PM Unterhaus_: do I have to change the permissions of the smp+affinity file?
01:51 PM pcw-home: no, you just change them as root
01:51 PM Unterhaus_: sudo not work?
01:52 PM pcw-home: sudo doesn't work with the echo command line (there may be another way)
01:52 PM Unterhaus_: okay
01:52 PM pcw-home: so I usually sudo -i
01:53 PM pcw-home: I do have a automated script for this mosmck gave me a while ago but have misplaced...
01:53 PM pcw-home: so sudo -i
01:54 PM pcw-home: cd /proc/ look at the interrupts file to find the Ethernet interrupt for the interface you are using
01:55 PM Unterhaus_: I did that
01:55 PM pcw-home: the cd to irq/#/
01:55 PM Unterhaus_: I think I changed it, machine hasn't crashed yet
01:55 PM Unterhaus_: does that change persist over a reboot?
01:56 PM pcw-home: no, which is where a script (and smarter people) come in
01:58 PM Unterhaus_: eh, if you aren't smart enough I'm in real trouble :)
02:04 PM Unterhaus_: I don't see a hm2_7i96.0.read.tmax only read.time
02:05 PM Unterhaus_ is now known as unterhausen
02:06 PM unterhausen: at least I got the affinity right
02:08 PM pcw-home: *tmax are parameters
02:10 PM pcw-home: Of course the "error finishing read" might be unrelated to latency... 5V power supply issues will show up this way
02:11 PM pcw-home: ( you can tell though by the LEDs, a red LED indicates loss of communications, but a power dropout will reset the card so no red LED )
02:11 PM unterhausen: it had a red led
02:12 PM unterhausen: I should put a better psu on it though
02:13 PM unterhausen: tmax is 830000
02:14 PM pcw-home: so OK so far...
02:14 PM unterhausen: I'll get back to you on thursday
02:15 PM pcw-home: usually things like videos cause the worst issues
02:16 PM pcw-home: ( note: tmax is in CPU clock periods on x86 )
02:17 PM pcw-home: so that's probably about 300 usec ( default timeout would be 800 usec )
02:19 PM unterhausen: so not too bad then
02:20 PM pcw-home: so far...
02:24 PM unterhausen: apparently I need to do something to defeat irqbalance
02:40 PM Centurion_Dan1 is now known as Centurion_Dan
03:02 PM roycroft: a well-yielded bat'leth is a formidable weapon that can defeat almost any foe
03:24 PM JT-Shop: after fingering out that the Bluewing has a left front and right front brake pad set the right side has shipped
03:25 PM JT-Shop: now if the battery and multi-function display cover would ship it will prob start snowing
03:27 PM Centurion_Dan1 is now known as Centurion_Dan
04:50 PM Centurion_Dan1 is now known as Centurion_Dan
05:13 PM JT-Shop: never bring a bat'leth to a gun fight
05:27 PM roycroft: unless you're a klingon
05:27 PM roycroft: in which case you still have the advantage
05:45 PM Centurion_Dan1 is now known as Centurion_Dan
06:09 PM unterhausen: i'm a bit surprised the information about smp affinity is so sparse
06:10 PM unterhausen: I wonder if anyone on the dev channel has a script
06:11 PM unterhausen: interesting unanswered question on stackexchange
06:18 PM Tom_L: saw this on kernel org https://www.kernel.org/doc/Documentation/IRQ-affinity.txt
06:19 PM Tom_L: https://docs.kernel.org/core-api/irq/irq-affinity.html
06:20 PM Tom_L: same thing just fancied with html
06:28 PM unterhausen: that's the part I know, I want to make it persist over reboots
06:31 PM Tom_L: https://www.kernel.org/doc/html/v4.19/admin-guide/kernel-parameters.html
06:31 PM Tom_L: for the brave... or ignorant
06:32 PM Tom_L: 6.xx may vary some
06:33 PM Tom_L: unterhausen, have you set /proc/irq/default_smp_affinity?
06:34 PM unterhausen: that kernel org link is amazing. there should be contests on how long you can make your kernel args
06:34 PM Tom_L: hahaha
06:34 PM Tom_L: would save running the gui is all i think
06:35 PM unterhausen: Tom, I was wondering if it was as simple as setting the default
06:35 PM Tom_L: that water is too deep for this swimmer
06:35 PM unterhausen: I been scrolling and didn't get to irq affinity yet
06:36 PM unterhausen: I finally got there and it says the argument is a cpu list, as described above
06:36 PM Tom_L: Some kernel parameters take a list of CPUs as a value, e.g. isolcpus, nohz_full, irqaffinity, rcu_nocbs. The format of this list is:
06:36 PM Tom_L: <cpu number>,...,<cpu number>
06:40 PM unterhausen: I wonder if I can set all 40 interrupts
06:40 PM Tom_L: trying to win the contest?
06:40 PM unterhausen: it wouldn't be that long, but it would be a good first step in the contest
06:41 PM unterhausen: the argument to irq affinity should be a mask
06:41 PM Tom_L: you should talk to memleak about kernel tuning
06:42 PM unterhausen: Is that Alec?
06:42 PM Tom_L: i'm not sure but i think so
06:43 PM unterhausen: RTAI guy?
06:43 PM Tom_L: yes
06:46 PM unterhausen: there are definitely some irq's I would like to keep off the lcnc cpus
06:47 PM Tom_L: http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux
06:47 PM Tom_L: ^^ dunno
06:51 PM unterhausen: I wonder what I did wrong back when I tried to do this before
06:51 PM Tom_L: take good notes
06:52 PM synfinatic_ is now known as synfinatic
06:54 PM unterhausen: I was just thinking I should have been taking notes, but I didn't learn much yet
06:58 PM unterhausen: this is interesting https://www.suse.com/support/kb/doc/?id=000017528
07:00 PM Tom_L: https://github.com/NTULINUX/RTAI
07:00 PM Tom_L: that's his site
07:03 PM unterhausen: the cpusets people are screwing up the isolcpu information
07:04 PM Tom_L: 12 doesn't have cset in that folder
07:04 PM Tom_L: not by default anyway
07:05 PM unterhausen: not applicable to us, I guess
07:05 PM unterhausen: there is some arguing over isolcpu vs cpuset
07:05 PM Tom_L: or done differently
07:06 PM Tom_L: that's one thing i don't like abou linux
07:06 PM Tom_L: changing ways things are done
07:06 PM Tom_L: how's some poor noob sap supposed to figure anything out
07:08 PM unterhausen: I'm not a newb, but they changed the way initialization was done and I've never forgiven them
07:08 PM Tom_L: there is a light swithch to turn on the light... how you throw the switch is up for grabs
07:08 PM unterhausen: and now ifconfig is no more
07:08 PM Tom_L: now what?
07:08 PM unterhausen: I finally got so I automatically typed ifconfig, rather than the more obvious ipconfig, and now they screwed that up
07:09 PM unterhausen: isn't it just "ip" now?
07:09 PM Tom_L: ahh..
07:09 PM Tom_L: yeah
07:10 PM Tom_L: somebody's always gotta stir the pot
07:10 PM unterhausen: of course, you can't just type ip, it has to have arguments
07:11 PM unterhausen: it's the little pointless changes in linux that make me wonder about my life choices.
07:11 PM Tom_L: how do you list the IPs?
07:14 PM roycroft: apt -y install net-tools; ifconfig -a :)
07:14 PM Tom_L: :)
07:15 PM unterhausen: good idea
07:15 PM roycroft: i generally don't mind deprecating utilities as long as there is a reasonable transition period AND the replacement is improved over the original
07:16 PM unterhausen: argument :) invalid
07:16 PM roycroft: so far i've not found that to be the case with ip(8)
07:18 PM unterhausen: somehow forgot I wasn't hooked up to the internet
07:18 PM Tom_L: the plug was next to solarwind's collet
07:19 PM unterhausen: my ability to lose things is second to none
07:20 PM unterhausen: actually, I don't know why I'm not hooked up to the internet, seems to be plugged in
07:24 PM unterhausen: net tools didn't install for some reason and the error messages are unhelpful
07:24 PM roycroft: it's net-tools
07:24 PM roycroft: with a hyphen and no spaces
07:25 PM roycroft: but try "apt update" first
07:25 PM roycroft: and be sure to do it as root
07:25 PM Tom_L: what's -a for?
07:25 PM Tom_L: not understood...
07:25 PM roycroft: in the ifconfig command?
07:26 PM unterhausen: it doesn't like my kernel config.
07:26 PM roycroft: it displays all interfaces
07:26 PM Tom_L: unable to locate package ifconfig
07:26 PM roycroft: there is no ifconfig package
07:26 PM unterhausen: not configured to use gzip compression
07:26 PM roycroft: ifconfig is part of net-tools
07:26 PM -!- #linuxcnc mode set to +v by ChanServ
07:26 PM unterhausen: it seems to need initramfs-tools which doesn't like my kernel
07:27 PM Tom_L: well skip that..
07:27 PM roycroft: yes
07:27 PM unterhausen: net-tools doesn't install
07:27 PM roycroft: it's already gotten weird enough that it's not worth doing
07:27 PM roycroft: now you get to learn how to use ip(8)
07:28 PM Tom_L: wtf is iproute2?
07:28 PM unterhausen: I ain't doin' no more learnin' tonight
07:28 PM Tom_L: and package manager shows net-tools in the list...
07:28 PM unterhausen: i think I'll just learn to type ip addr
07:29 PM unterhausen: which is ifconfig
07:29 PM Tom_L: what if you're not sure of the ip?
07:29 PM unterhausen: addr is a string
07:29 PM unterhausen: not an actual address
07:30 PM Tom_L: k
07:30 PM roycroft: alias ifconfig "ip addr"
07:30 PM roycroft: there you go :)
07:30 PM unterhausen: not a bad idea
07:30 PM Tom_L: ip mac is a thing too
07:30 PM unterhausen: because I'm going to forge
07:30 PM unterhausen: I'll forget, but probably forge as well
07:33 PM roycroft: is ip mac like arp -a?
07:33 PM Tom_L: dunno
07:33 PM Tom_L: add set del are the options
07:33 PM Tom_L: and it's ip macset
07:35 PM unterhausen: maybe just turning off the irq_balancer daemon is enough. CPU0 gets all the irqs
07:35 PM unterhausen: and then change the smp affinity for the ethernet card
07:36 PM unterhausen: I give up for now
09:41 PM solarwind: What's the advantage of using roughing end mills? The ones with rough bumpy edges?
09:42 PM lcnc-relay: <zincboy_ca_on> Chip control mostly.
09:43 PM lcnc-relay: <zincboy_ca_on> When roughing I would get a massive pile of long chips without a rougher.
09:44 PM XXCoder: large areas cut away much faster
09:44 PM XXCoder: still need to finish it but yea
09:44 PM XXCoder: chip control is also great like zinc said
09:45 PM Tom_L: solarwind, more edges to cut yourself on
09:46 PM solarwind: Ok yeah the chips are much smaller
09:46 PM solarwind: makes sense
09:47 PM Tom_L: http://tom-itx.no-ip.biz:81/~webpage/cnc/cutters/Carbide_rougher4.jpg
09:47 PM XXCoder: tool life is great also
09:47 PM XXCoder: as long as its properly cooled etc
09:48 PM Tom_L: we found some time back on a large part with lots of pocets it was quicker to plunge cut holes then remove the rest than trying to rough mill them out
09:49 PM solarwind: Tom_L that's a nice one
09:49 PM lcnc-relay: <zincboy_ca_on> I don't find there is a practical difference in how fast you can remove material. Technically you should be able to feed a rougher more aggressively. I could double my MRR but it would mean feeding at 200ipm instead of 100ipm (1" DOC, 0.05" WOC) and my machine doesn't accelerate fast enough to really push 200 on anything other than trivial stuff.
09:50 PM solarwind: I'm finding M42 tool steel works very well when being cooled with an air blow gun
09:50 PM Tom_L: get a mister and use it dry then you don't have to stand there with a blow gun
09:50 PM solarwind: Yeah that's how I have it right now
09:51 PM Tom_L: http://tom-itx.no-ip.biz:81/~webpage/cnc/cutters/Carbide_rougher2.jpg
09:51 PM Tom_L: i like lakeshore cutters
09:51 PM Tom_L: between them and online carbide
09:51 PM lcnc-relay: <zincboy_ca_on> I have seen some TiAlN coated HSS mills. Anyone tried one of those? I can't see how it would work well since the TiAlN likes to be hot.
09:52 PM Tom_L: zincboy_ca_on, TiAlN is for steel right?
09:52 PM lcnc-relay: <zincboy_ca_on> Yes. best used dry on steel.
09:52 PM Tom_L: i have a handfull i tried on my little mill
09:53 PM Tom_L: zrn is alum iirc
09:53 PM solarwind: I have TiAlN end mills, definitely don't want them hot
09:53 PM solarwind: I can't think of _anything_ that runs better hot
09:54 PM Tom_L: even using zrn on alum i still get a better finish with coolant
09:54 PM solarwind: yeah I try to keep everything as cold as possible and I get very good tool life
09:54 PM solarwind: if I let things get hot, it ruins the edge fast
09:54 PM Tom_L: http://tom-itx.no-ip.biz:81/~webpage/cnc/cutters/60deg_3flute2.jpg
09:54 PM Tom_L: those are all TiAlN
09:54 PM Tom_L: hi helix
09:55 PM XXCoder: only some tooling likes it hot
09:55 PM XXCoder: cermics usually
09:55 PM lcnc-relay: <zincboy_ca_on> TiAlN is best run hot with no coolant. The coating actually works best that way. Coolant just causes micro cracking and failure of the edge.
09:56 PM Tom_L: zincboy_ca_on, i heard that only happens if you go from dry to wet etc
09:56 PM Tom_L: if you start with coolant keep it on
09:56 PM lcnc-relay: <zincboy_ca_on> Same for coolant unless your flood coolant means it.
09:56 PM solarwind: I can understand about the micro cracking
09:56 PM solarwind: there are indeed localized high heat zones right at the edge
09:56 PM roycroft: that's the case with carbide tooling, tom_l, but i don't know if it's the case with tialn
09:57 PM solarwind: carbide isn't as sensitive as you make it out to be
09:57 PM Tom_L: it is brittle though
09:57 PM solarwind: air blast helps, I've had zero issues with cracking
09:57 PM Tom_L: we cut tungsten with carbide
09:57 PM solarwind: only during interrupted heavy cut if you use the wrong grade
09:57 PM Tom_L: wing ballast
09:57 PM solarwind: the tough grades are very crack resistant
09:58 PM lcnc-relay: <zincboy_ca_on> TiAlN is just a coating. Airblast is mostly for chip clearance. Recutting chips will cause rapid failure as well.
09:58 PM solarwind: there is a huge difference between the grades in terms of brittleness, transverse rupture strenght, etc.
09:58 PM XXCoder: made parts using cobolt tool. it was interesting. very expensive though ouch
09:59 PM Tom_L: http://tom-itx.no-ip.biz:81/~webpage/cnc/cutters/Carbide_em2.jpg
09:59 PM Tom_L: i _am_ curious what that brown coating is though
09:59 PM Tom_L: i know it was intended for Ti
10:00 PM XXCoder: box dont say it?
10:00 PM Tom_L: got them as rejects from kids work
10:00 PM XXCoder: usually it would state what coating it has
10:00 PM XXCoder: hmm
10:01 PM Tom_L: notice they've all been used in shrink holders
10:02 PM XXCoder: strunk holders. excellent but I hate changing tools on those lol
10:02 PM Tom_L: the discolored shank gives it away
10:02 PM lcnc-relay: <zincboy_ca_on> Might be TiCN but it is hard to tell by color alone.
10:02 PM Tom_L: yeah
10:02 PM Tom_L: that was the only one like that
10:02 PM solarwind: My spindle has about 10µm of runout. Might be time for a regrind/bearing replacement
10:03 PM XXCoder: you sure its not misalign?
10:03 PM solarwind: possibly, have to check
10:05 PM XXCoder: other thing to check is chips or impression from chip on tapers if you are using cat or similiar toolholder
10:05 PM roycroft: so my parcel that took the scenic route down to socal and back is still not here
10:05 PM roycroft: it's been out for delivery two days in a row now
10:05 PM roycroft: with an alert last night that it was not delivered
10:29 PM roycroft: well, a very grumpy postal worker just delivered my vise screw
10:29 PM roycroft: and it is 3tpi, so that's great - i was afraid it might be 5tpi, which is what most of this type are
10:30 PM roycroft: so now i can get working on that project
10:30 PM roycroft: the wooden screw one
10:50 PM CloudEvil: :)
10:51 PM CloudEvil: It would be nice if you could do proper grain orientation alas I think that's impossible
10:52 PM XXCoder: you probably can twist wood, then mill it
10:52 PM CloudEvil: I was meaning twist it so the grain is along the thread
10:52 PM XXCoder: anyway its very possible. earliest screws was made from wood, and famous print press was made using wood screw
10:52 PM CloudEvil: As it grows
10:52 PM roycroft: it will be strong enough
10:53 PM roycroft: i'll use a dense hardwood
10:53 PM CloudEvil: It's quite possible to do it regularly, I've done it.
10:53 PM roycroft: and the screws i'm making first will be 2-1/2" in diameter
10:53 PM CloudEvil: - dowel and chip carving
10:54 PM roycroft: i don't have drawings yet of what i'll be doing, but i can describe it
10:54 PM roycroft: i just got an acme thread vise screw and nut
10:54 PM roycroft: it's 3tpi, the thread pitch i want for my wooden screws
10:54 PM roycroft: for the 2-1/2" ones
10:55 PM roycroft: on the end of the screw i'll attach a 2" or so cylinder that will hold a cutter
10:55 PM roycroft: kind of like a boring bar
10:55 PM roycroft: then i'll insert the cutter with the correct thread profile for the wooden screw
10:56 PM roycroft: i'll use the acme screw to index and advance the cutter into the block of wood that i'll use as the nut
10:56 PM CloudEvil: How long are you aiming at?
10:56 PM roycroft: for the screw i think i'll use a live cutter
10:56 PM CloudEvil: live cutters are the way.
10:56 PM roycroft: well the not will only have to be 2"-3" thick
10:57 PM roycroft: i can use that same acme screw to index the blank that i'll use to make the wooden screw
10:57 PM roycroft: i'll set up a router with the appropriate cutter, and feed the blank into the router cutter with the acme screw
10:58 PM roycroft: i think i can make it work very well, and very efficiently
11:00 PM roycroft: i have a similar 5tpi vise screw already, which i can use to make smaller screws
11:00 PM XXCoder: I still remember that video that showed how to make first inside threads
11:01 PM XXCoder: it was pretty amazing
11:02 PM roycroft: a live cutter for the inside threads would be nice, but i am not set up to do that right now
11:03 PM roycroft: i could make a mechanical machine to do it - i would not need a cnc mill
11:03 PM roycroft: but it would be a lot of work, and i'd rather try this method first
11:04 PM XXCoder: yeah.
11:04 PM roycroft: dems hold the senate in virginia