#avr Logs

Apr 30 2021

07:16 AM specing_ is now known as specing
01:40 PM Kabriel: I would like to send binary data over the usart. I was looking into how to frame the data. The best option I have come across is COBS. Is this pretty standard, or are there other, better framing methods?
01:42 PM cehteh_: huh? anything that floats your boat
01:43 PM cehteh_: i'd prefer text protocols if possible, allow for easier debugging, but need more bandwidth and possibly more cpu cycles
01:44 PM cehteh_: with uart you can just echo back all data and verify that, or add checksums of some sort
03:37 PM vmt: i don't quite appreciate the way scripting languages and what not kids use nowadays lead to some make-believe distinction between "text" and "binary" data
03:38 PM vmt: well, that's not exactly what i was gunning for. point being data is data and the representation is whatever you want it to be
03:51 PM exp: binary data still needs framing
03:51 PM exp: copying structs blindly across a usart is a recipe for pain
03:51 PM exp: you need sync, so you can pick up a stream midway
03:51 PM exp: you need CRCs or some sort of PEC, so you can validate received data
03:51 PM exp: you need a backoff algorithm if you're half duplex
03:52 PM exp: the typical distinction between 'text' and 'binary' is that text is line buffered, which is a perfectly valid abstraction for unidirectional serial
03:52 PM exp: and Kabriel your biggest problem with COBS will likely be its memory requirements
04:45 PM vmt: *data* needs framing.
04:54 PM exp: right, but line delimited text is a form of framing, that is a valid thing to distinguish
04:56 PM zgu: and with binary framing your data can often include the framing characters, so you have to be more creative for sync detection
04:57 PM exp: at my job we have some naive developers who effectively just send structs over a tcp connection, to a machine running on a totally different architecture, with absolutely zero error detection
04:58 PM vmt: there's nothing that says you can't have "delimited" binary data as well
04:58 PM vmt: binary should be in quotes as well, really
04:58 PM exp: well the main characteristic of 'binary' in this sense is that chars 0-255 are valid
04:58 PM exp: so you can't simply delimit it
04:59 PM vmt: your data can be whatever you want
04:59 PM vmt: and so can its representation
04:59 PM exp: right but i believe that's what Kabriel was asking, how best to represent their data
04:59 PM exp: and the answer is that COBS is perfectly valid if a little heavy due to the 256 bytes or whatever it is it requires
04:59 PM zgu: at my old job we did that. but it was all windows talking to windows so it didn't usually matter
05:00 PM vmt: right. well, anything goes at that point
05:00 PM exp: zgu: they don't even version things properly, and they have assumed a header file transmits all of the information required to decode these structs
05:00 PM exp: but without explicitly calling out or ensuring alignment, or the size of ints or anything
05:00 PM exp: it has been... frustrating
05:01 PM zgu: yeah i remember one time we had to add a field to the end of one of the messages, and the other end wouldn't read the full packet right if it was an older version
05:01 PM vmt: if you need a lot of flexibility, an underlying length-payload is probably the way to go
05:01 PM vmt: with a predefined endianness etc.
05:01 PM zgu: yeah
05:02 PM vmt: and a fixed length for the `length`
05:02 PM exp: i agree 2 or 4 byte length field is appropriate, but you also need a way to sync on that
05:03 PM vmt: what do you mean by syncing?
05:03 PM zgu: header with one or two fixed bytes (if you're clever you can use that for endian detection) and length, followed by inner layers with their own headers
05:03 PM exp: vmt: i mean if you miss a few bytes, you need to be able to locate which of the following bytes are the packet length
05:03 PM exp: as otherwise you won't be able to distinguish "a" from a 64 byte packet, for example
05:03 PM vmt: if you miss a few *bytes* you're probably fakked anyway
05:03 PM exp: I believe COBS uses nulls for that, and ensures there's no nulls elsewhere
05:04 PM exp: it's also valid to just repeat specific characters, so 0 becomes 00, and you can then use 0101 or whatever you like for sync
05:04 PM exp: and nah lots of cases where you can be blocked and miss bytes on a usart or anything like that
05:04 PM zgu: kinda like bit stuffing but at byte level. or \ escaping
05:05 PM exp: most everything is packetised nowadays and you definitely can lose packets just due to emi
05:05 PM exp: zgu: same concept indeed
05:05 PM vmt: yeah, but on that frame
05:05 PM vmt: but yeah i got you now, that's what you were kind of syaing
05:05 PM exp: well, it's what makes it a frame rather than just a stream
05:06 PM exp: https://en.wikipedia.org/wiki/Syncword
05:06 PM vmt: safeword
05:07 PM exp: my safe word is 0x6C6F6C6869
05:08 PM Kabriel: Correct, using "strings" as the frame, you limit the actual binary data that is sent, so you can use characters to delimit the end of a message for example (like \r or whatever).
05:09 PM Kabriel: This puts more burden on the receiving end to do whatever deserialization is required to get the actual data you want, be it a structure or simple some number.
05:10 PM vmt: well, but similarly you *can* have a boundary for any data.
05:10 PM Kabriel: exp: how does COBS use so much memory? I was thinking it seems to only increase the size if you have a lot of the "delimiting" characters (e.g. most of the time 0x00 is used).
05:11 PM vmt: what you probably really want, is to try to avoid sending arbitrary strings as much as possible
05:11 PM Kabriel: vmt: but what if you pick up reading in the middle of a message?
05:12 PM Kabriel: I actually want to remove all the string processing that is in the current usart communication protocol and move to raw binary protocol.
05:12 PM vmt: well you'll need some kind of sync, as already mentioned
05:13 PM vmt: mind you, sure, it can also be \n
05:13 PM vmt: Kabriel: yes but what i meant, sending as little text as possible in the payload
05:15 PM exp: Kabriel: I only had a quick read over, but iirc it requires up to 256 byte buffer
05:15 PM exp: which isn't much memory, but it's plenty on little AVRs
05:16 PM vmt: anyway, my 2 cents is that figure out wtf data you want to send, and go from there
05:17 PM vmt: i.e. solve that problem first, and don't create a new one before solving the first
05:19 PM Kabriel: the general idea with COBS is that the stream never sends a 0x00 except at the end of a frame. So you can *always* know the beginning of a new frame.
05:19 PM cehteh_: how much data? always the same data with constant frame length? will the usart be statuated for longer times? what latency can you accept? how do you want to syncronize? bidirectional request/reply? etc ... many questions
05:19 PM vmt: Kabriel: well, that would be your sync then
05:19 PM Kabriel: it is a req-rep model, messages vary in length, but always short.
05:19 PM cehteh_: serial works craps when you satuate the link
05:19 PM vmt: Kabriel: but i would suggest you figure out your data first, before you try to figure out your method
05:20 PM cehteh_: why not text?
05:20 PM cehteh_: i almost always have a cli like interface on the serual where i can enter commands manually
05:21 PM cehteh_: when the command buffer is empty then the firmware sends sesnor data once a second or so
05:21 PM vmt: i almost always have the 3d file viewer from jurassic park and i can go all "it's a unix system" for debugging
05:21 PM Kabriel: cehteh_: speed and remove all the text processing, which is not needed. Text representation is limited and is for humans to read.
05:22 PM vmt: Kabriel: but this really depends on the data
05:22 PM cehteh_: Kabriel: well i know nothing about what you want to do, but text representation when well done is unambigous, it has some framing and parsing it wont cost much in most cases
05:23 PM cehteh_: when you send telemetry data of a spaceship flight controller at 1000hz of course then text may have some problems
05:23 PM cehteh_: so it really depends
05:24 PM Kabriel: vmt: I suppose if you are sending text messages to the mcu, then text is the way to go. I'm sending lots of numbers, and the text representation of numbers has lots of overhead, compared to sending binary.
05:24 PM Kabriel: Think about other protocols (TCP for example); they didn't use text for a reason.
05:25 PM cehteh_: saying 'text' is for humans is a bit misleading, because there are a lot text protocols out there which are not really meant for humans to read (one could read html or xml or json, but thats not theiur primary use case, they are made for machine digestion)
05:26 PM cehteh_: anyway .. your project, your choice
05:27 PM Kabriel: Well, the real question was, is anyone using something better than COBS for data framing and it sounds like, at least those who have spoken up thus far, use strings for data framing.
05:28 PM vmt: your framing can be whatever. use a "syncword" to mark the beginning of each frame.
05:28 PM cehteh_: note that it is not that easy ... on a satuated serial link you wont even be able to synchonize to a symbol start
05:28 PM vmt: and you have to handle these cases appropriately
05:28 PM Kabriel: If anyone is interested, I came across a few articles that talk about this.
05:29 PM Kabriel: Here is one using an escape sequence: https://eli.thegreenplace.net/2009/08/12/framing-in-serial-communications/
05:29 PM cehteh_: and strings is basically using \n as sync marker
05:29 PM vmt: i can provide you with more consulting for a meager $50/hr
05:29 PM vmt: yes
05:29 PM Kabriel: And here is one using COBS byte suffing https://www.embeddedrelated.com/showarticle/113.php
05:29 PM cehteh_: doh vmt is cheap :D
05:29 PM vmt: yes
05:30 PM vmt: but first five hours we will participate in meditative yoga
05:30 PM cehteh_: ah ok :)
05:30 PM cehteh_: as usual
05:31 PM exp: Kabriel: can you afford to lose numbers?
05:31 PM cehteh_: anyway .. first lay out the requirement you really have, is it one sensor data word per second? is it an image grabbed from a camera ..
05:32 PM cehteh_: deciding on the encoding/transmission comes after you know the requirements
05:32 PM exp: you mentioned it's request / response, so all you require for that is 'ready to send' signal, ala primitive serial signalling
05:32 PM Kabriel: Yes. I'm not interested in sequencing methods or anything like that ... way too much overhead. COBs didn't seem that bad, but I'm going to read the paper by Cheshire and Baker tonight.
05:32 PM Kabriel: ^ exp: Yes I can afford to lose a message
05:32 PM cehteh_: binary isnt totally wrong, i'D just suggest to go as far as comfortable with text.
05:32 PM exp: Kabriel: if you can afford to lose data, it's request/response, and you have some internal timeout to avoid deadlocking, just end the packet with a known sequence that can't exist in it otherwise
05:33 PM exp: not quite COBS but no readahead required
05:33 PM exp: if you're truly sending data where every byte sequence is possible, then use the lowest overhead COBS like mechanism you're happy with
05:34 PM Kabriel: I thought about a magic number or sequence, but I'm dealing with lots of numbers, so there is the off chance that the binary data would contain that sequence.
05:34 PM exp: the one thing you haven't mentioned and you need to do is a PEC or CRC etc
05:34 PM exp: so you'll need some primitive framing to encompass this
05:34 PM Kabriel: exp: that was a follow on question, depending on the currenct conversation.
05:35 PM exp: I don't really think there's much more advice that can be offered, COBS is a fine model, and it's clear you understand the problems involved, the only thing i'll say is when you design your frame format, for the love of god include a version tag of some description :)
05:35 PM Kabriel: I have a working system that is string based now. I'm looking to design a better (faster and more efficient) data protocol, to reduce the overhead in the string based system.
05:36 PM Kabriel: exp: of course!
05:37 PM cehteh_: you can just send size, data...., crc16 since it is request/respose the receiver can ignore anything until it did a request, maybe add a sequence number and fine
05:37 PM cehteh_: in case of crc fail you can request a retransmit
05:38 PM cehteh_: (or drop the invalid data and go on with the next) .. whatever
05:38 PM exp: yeah don't start adding retransmits
05:38 PM exp: that requires a sending buffer, and that requires acks, and soon enough you've reinvented tcp but worse
05:38 PM cehteh_: depends on whats required
05:40 PM cehteh_: when you dont know the size in advance then you may need some encoding that has a marker and that marker needs to be 'escaped' when present in the datastream ... COBS does that but there are plenty other ways to do that
05:40 PM Kabriel: What are the opinions on the various CRC computations in libavr (util/crc16.h)?
05:40 PM cehteh_: they work
05:41 PM cehteh_: they have a useable interface and are pretty well optimized
05:41 PM cehteh_: i use it for my eeprom configuration storage
05:41 PM Evidlo: what are people's thoughts on the new AVRs and their debugWire compared to the old ones? I help run a design course and we're trying to standardize our guides around one family, probably atmega328
05:42 PM exp: i think: have you bought the chips ahead of time? ;-)
05:42 PM exp: i've just switched to an xmega series so looking forward to using PDI
05:42 PM Evidlo: we haven't bought anything
05:43 PM cehteh_: Kabriel: what protection level against malformed data do you need?
05:43 PM exp: Evidlo: the joke is that there's a lot of parts out of stock currently
05:44 PM cehteh_: crc16 is 16 bit which means theoretically/on average one in 65536 errors may go undetected ... actually its a bit better because it is desigend to catch common transmission errors, but something along that is what it can offer
05:45 PM Kabriel: cehteh_: right now, even with the string based system, there is *no* check, which I consider bad.
05:45 PM Kabriel: If messages are small (e.g. <50 bytes) is crc8 good enough ... that would be my question.
05:45 PM cehteh_: thats why i saied just echo back any data can be a viable way to ensure correctness to (when you check the echoed data) ... it would be extremely rare that a transmission error happens in the same bit in both directions
05:45 PM exp: only you can define 'good enough'
05:46 PM cehteh_: in some cases just xor is good enough ... in others you want a message authentication digest of 512 bits
05:47 PM cehteh_: (which wont work so well on AVR's ;D)
05:48 PM Kabriel: ... I'm late for dinner. Thanks for the discussion; I'll read up on CRC tomorrow.
05:48 PM exp: have fun
05:48 PM cehteh_: how much errors to you expect (ling cables? noisy environment?) and what will happen when the data gets corrupted? hosue on fire, nuclear power plant gone boom, your garden watering wont cut off the sprinkler for 1 second?
05:49 PM exp: everything cehteh_ is saying is correct and important to consider
05:50 PM exp: are you transmitting something that will be used in a legal record? or are you transmitting the temperature of your heated seeding tray?
05:50 PM exp: speaking of which, I have some 433mhz OOK to decode :)
06:25 PM vgtw_ is now known as vgtw
07:18 PM specing_ is now known as specing