Why can't ESP32 read all of serial incoming data?

Question

I am trying to change configration settings of E70 LoRa module. It works with AT commands. Everything is working normally, but I can't read all incoming data; after the "09:25:01.008 -> TFOCFG Out" line the ESP32 stops reading.

As you can see in the datasheet (below image) there are some lines also. On the other hand, when I use an Arduino Nano instead of the ESP32, it reads distorted and the characters become strange symbols. How can I read properly?

#include <HardwareSerial.h>
#define M0 32  // 3in1 PCB mizde pin 7
#define M1 33  // 3in1 PCB mizde pin 6
#define RX 27  // Lora nın 3. pini RX in ESP 32 de hangi pine bağlı olduğu
#define TX 35  // Lora nın 4. pini TX in ESP 32 de hangi pine bağlı olduğu

    HardwareSerial fixajSerial(1);
    
    void setup() {
      Serial.begin(115200);
      fixajSerial.begin(115200, SERIAL_8N1, TX, RX);
      pinMode(M0, OUTPUT);
      pinMode(M1, OUTPUT);
      digitalWrite(M0, 1);  // E70 config mode
      digitalWrite(M1, 1);
    
      pinMode(14, OUTPUT);
    
      Serial.print("start");
      fixajSerial.print("++");
      delay(100);
      fixajSerial.print("+++");
      delay(100);
      fixajSerial.print("AT+HELP\n");
    }
    
    void loop() {
    
      if (Serial.available()) {
        fixajSerial.write(Serial.read());
      }
    
      if (fixajSerial.available()) {
        Serial.write(fixajSerial.read());
      }
    }

Output:

    09:25:00.792 -> startEnter AT Mode
    09:25:00.962 -> AT+HELP
    09:25:00.962 -> ATCommand           HELP                                              
    09:25:00.962 -> CLINFO              This Is Clean Node Net Info                       
    09:25:01.008 -> DINFO               This Is device Infomation Code                    
    09:25:01.008 -> WMCFG               Device Work Mode Config                           
    09:25:01.008 -> FHCFG               FH Config                                         
    09:25:01.008 -> TFOCFG              Out

First store read data (Serial.read()) in buffer and after that send buffer (fixajSerial.write(...)). — Gregory, Jan 18 '23 at 07:49
You're sending `++` then `+++` to the device, which is probably not helping. — Polynomial, Jan 18 '23 at 12:12
@Polynomial probably you are right but i see this note at docs so I made this. note is "AT instruction is configured, needs the serial port to send no more than two bytes data to wake upthedevice .After the wake-up byte is sent, the user needs to wait for 100ms to send the real data and the wake-up datawill bediscarded." — mehmet, Jan 22 '23 at 09:04

Polynomial · Answer 1 · 2023-01-18T14:54:52.867

2

This is a buffering issue. If you count up the bytes in the text you received, you get 256. The HardwareSerial class uses hardware receive buffers, and sets a default receive buffer size of 256.

You're writing commands in your setup() function, but you're not handling reads until you get to loop(), which isn't executed until setup() returns. As such, the buffer fills up and all other received data is lost.

You should set up a callback using onReceive before sending the commands, so you can handle the receive interrupt and read the serial buffer before it overflows.

You should also read all available bytes into a buffer, then write them back out, rather doing an availability check, one byte read, one byte write, and then again in the other direction. This will vastly reduce the time it takes to handle the serial data.

edited Jan 18 '23 at 14:54

answered Jan 18 '23 at 12:29

Polynomial

10,562
5
47
88

Are you sure? It looks to me like sending AT+HELP is the last thing in the setup function, so the loop function starts immediately after that. – user253751 Jan 18 '23 at 13:39
@user253751 Yes, but since it's reading one byte at a time from the input and writing it back to the output in a tight loop, and each call into the serial functions involves several lock acquisitions down the stack (one in the UART HAL, one or more in the underlying UART calls, for each of the six calls in the loop iteration, plus event queue operations and interrupts), it's probably taking too long to get the data out of the ring buffer. The easiest way to check would be to set up the `serialEvent1` callback and see what events are showing up there. – Polynomial Jan 18 '23 at 14:57
then the 256 number is irrelevant – user253751 Jan 18 '23 at 14:58
@user253751 Why would it be irrelevant? The FIFO fills up while the loop reads data. Once the FIFO fills up, all new data gets dropped. The code then reads the remainder of the FIFO, which gets you to 256 bytes. – Polynomial Jan 18 '23 at 15:34
and why does the loop wait for the FIFO to fill up before it takes data out of the FIFO? – user253751 Jan 18 '23 at 15:43
@user253751 It isn't waiting, it's just too slow to beat the FIFO filling up. – Polynomial Jan 18 '23 at 15:44
if it reads from the FIFO then you should get more than 256 characters because you should get 256 + the number read while it is filling – user253751 Jan 18 '23 at 15:45
that threw me off too (and I used the wrong terminology above) but after digging into the UART driver implementation it makes more sense. the hardware ring buffer is only fetched with `xRingbufferReceive` when the remaining size in the driver UART buffer is zero. this means that on the first read call, the UART receive buffer size is zero, so the buffer is fetched. subsequent reads then don't touch the FIFO at all, but instead read from the existing buffer. once the buffer is empty, it is released with `vRingbufferReturnItem`, and a subsequent read will grab another buffer from the FIFO. – Polynomial Jan 18 '23 at 16:21
however, if the FIFO fills up _before_ you get back around to returning the ringbuffer (note: handwaving some implementation details re: flow control, synchronisation with cross-core reads from tasks, etc.) it raises a FIFO full event _and resets the FIFO_. so you lose all the new data coming in. so you get to the end of the UART driver's buffer (which is 256 bytes) and then you get no more data. – Polynomial Jan 18 '23 at 16:24
and if the system is too slow to do this simple while(true) {if(canReceive()) receive():} then how will callbacks improve it? – user253751 Jan 18 '23 at 16:33
the callback is mostly just to ensure that reads are performed in a timely manner, and it's easier to architect the code to do that reliably during setup and loop when there's a callback. the more important part is to swap the single byte read/write out for a call that reads the entire available buffer, so it's not doing an `available()` call then a `read()` call then a `write()` call for every single byte, which is not only slow but also introduces critical section spinlock contention in the UART ISR if the loop task happens to be scheduled on the other core to the UART ISR. – Polynomial Jan 18 '23 at 16:47

score 0 · Answer 2 · answered Jan 18 '23 at 15:09

This is one of those problems that I have seen all too frequently. In short, you have the LoRa module sending serial data to you as fast as it can. At the same time, you are sending that data (at the same rate) to another serial device. What you are seeing is that it takes longer for the Arduino to transmit data to the serial device that it does for the LoRa to send data to the Arduino. Thus, assuming no limit on the amount of data the LoRa can transmit, eventually you will begin to lose data from the LoRa. There are two solutions to this problem, either have a receive buffer for the LoRa that is larger than the maximum amount of data it can transmit at one time, or use a higher baud rate for your serial device.

Why can't ESP32 read all of serial incoming data?

2 Answers2